[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 512153027 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003331] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.008415] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010012] pid_max: default: 32768 minimum: 301 [ 0.011128] LSM: Security Framework initializing [ 0.012059] Yama: becoming mindful. [ 0.014014] SELinux: Initializing. [ 0.015058] *** VALIDATE selinux *** [ 0.023276] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028325] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029152] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031075] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032105] *** VALIDATE tmpfs *** [ 0.034318] *** VALIDATE proc *** [ 0.035244] *** VALIDATE cgroup *** [ 0.036008] *** VALIDATE cgroup2 *** [ 0.037262] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038173] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040030] Spectre V2 : User space: Vulnerable [ 0.041007] Speculative Store Bypass: Vulnerable [ 0.044086] debug: unmapping init [mem 0xffffffffaba59000-0xffffffffaba60fff] [ 0.046165] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047665] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048024] ... version: 2 [ 0.049009] ... bit width: 48 [ 0.050011] ... generic registers: 4 [ 0.051011] ... value mask: 0000ffffffffffff [ 0.052011] ... max period: 00007fffffffffff [ 0.053013] ... fixed-purpose events: 3 [ 0.054009] ... event mask: 000000070000000f [ 0.055341] rcu: Hierarchical SRCU implementation. [ 0.057488] smp: Bringing up secondary CPUs ... [ 0.058518] x86: Booting SMP configuration: [ 0.059018] .... node #0, CPUs: #1 #2 #3 [ 0.062259] smp: Brought up 1 node, 4 CPUs [ 0.064010] smpboot: Max logical packages: 1 [ 0.065018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.143622] node 0 deferred pages initialised in 75ms [ 0.147137] devtmpfs: initialized [ 0.148226] x86/mm: Memory block size: 128MB [ 0.150821] gcov: version magic: 0x41383552 [ 0.152034] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.153127] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.155281] pinctrl core: initialized pinctrl subsystem [ 0.156240] [ 0.156699] ************************************************************* [ 0.157012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.158018] ** ** [ 0.159012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.160014] ** ** [ 0.161067] ** This means that this kernel is built to expose internal ** [ 0.162013] ** IOMMU data structures, which may compromise security on ** [ 0.163013] ** your system. ** [ 0.164014] ** ** [ 0.165013] ** If you see this message and you are not debugging the ** [ 0.166010] ** kernel, report this immediately to your vendor! ** [ 0.167014] ** ** [ 0.168012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.169007] ************************************************************* [ 0.170734] NET: Registered protocol family 16 [ 0.171478] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.172057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.173060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.174662] cpuidle: using governor menu [ 0.175711] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.179309] PCI: Using configuration type 1 for base access [ 0.182183] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.192560] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.193066] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.194313] cryptd: max_cpu_qlen set to 1000 [ 0.197243] ACPI: Added _OSI(Module Device) [ 0.199011] ACPI: Added _OSI(Processor Device) [ 0.202014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.204010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.209637] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.216498] ACPI: Interpreter enabled [ 0.217054] ACPI: PM: (supports S0 S3 S4 S5) [ 0.218010] ACPI: Using IOAPIC for interrupt routing [ 0.219102] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.220395] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.229346] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.230034] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.231022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.232082] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.234254] acpiphp: Slot [2] registered [ 0.235081] acpiphp: Slot [5] registered [ 0.236102] acpiphp: Slot [6] registered [ 0.237077] acpiphp: Slot [7] registered [ 0.238100] acpiphp: Slot [8] registered [ 0.239077] acpiphp: Slot [9] registered [ 0.240100] acpiphp: Slot [10] registered [ 0.241452] acpiphp: Slot [3] registered [ 0.242078] acpiphp: Slot [4] registered [ 0.243114] acpiphp: Slot [11] registered [ 0.244123] acpiphp: Slot [12] registered [ 0.245088] acpiphp: Slot [13] registered [ 0.246072] acpiphp: Slot [14] registered [ 0.247073] acpiphp: Slot [15] registered [ 0.248074] acpiphp: Slot [16] registered [ 0.249076] acpiphp: Slot [17] registered [ 0.250074] acpiphp: Slot [18] registered [ 0.251089] acpiphp: Slot [19] registered [ 0.252068] acpiphp: Slot [20] registered [ 0.253071] acpiphp: Slot [21] registered [ 0.254070] acpiphp: Slot [22] registered [ 0.255077] acpiphp: Slot [23] registered [ 0.256078] acpiphp: Slot [24] registered [ 0.257120] acpiphp: Slot [25] registered [ 0.258077] acpiphp: Slot [26] registered [ 0.259079] acpiphp: Slot [27] registered [ 0.260083] acpiphp: Slot [28] registered [ 0.261078] acpiphp: Slot [29] registered [ 0.262081] acpiphp: Slot [30] registered [ 0.263073] acpiphp: Slot [31] registered [ 0.264000] PCI host bridge to bus 0000:00 [ 0.264000] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.266024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.268053] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.270018] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.273020] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.276039] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.278165] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.281057] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.284396] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.294717] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.299009] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.302019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.305015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.307015] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.309612] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.312680] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.315037] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.318844] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.325012] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.337011] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.342011] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.348884] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.357012] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.369015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.387016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.439497] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.454015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.461013] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.478013] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.485959] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.493016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.505045] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.527017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.546988] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.561016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.567015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.583015] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.593927] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.611017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.620016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.645017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.661518] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.674013] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.688021] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.718017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.737172] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.739289] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.742332] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.744315] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.746196] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.751212] iommu: Default domain type: Passthrough [ 0.754344] SCSI subsystem initialized [ 0.755137] ACPI: bus type USB registered [ 0.757062] usbcore: registered new interface driver usbfs [ 0.758043] usbcore: registered new interface driver hub [ 0.760093] usbcore: registered new device driver usb [ 0.761089] pps_core: LinuxPPS API ver. 1 registered [ 0.763010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.766061] PTP clock support registered [ 0.769080] EDAC MC: Ver: 3.0.0 [ 0.771178] PCI: Using ACPI for IRQ routing [ 0.772867] NetLabel: Initializing [ 0.774009] NetLabel: domain hash size = 128 [ 0.776008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.778080] NetLabel: unlabeled traffic allowed by default [ 0.780150] vgaarb: loaded [ 0.781241] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.783013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.789286] clocksource: Switched to clocksource kvm-clock [ 0.894500] VFS: Disk quotas dquot_6.6.0 [ 0.896072] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.898390] *** VALIDATE ramfs *** [ 0.899310] *** VALIDATE hugetlbfs *** [ 0.900499] pnp: PnP ACPI init [ 0.905365] pnp: PnP ACPI: found 6 devices [ 0.923727] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.927119] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.929483] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.931700] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.934322] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.936933] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.939846] NET: Registered protocol family 2 [ 0.942397] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.947103] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.950612] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.955835] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.959381] TCP: Hash tables configured (established 65536 bind 65536) [ 0.962425] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.965422] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.968451] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.971552] NET: Registered protocol family 1 [ 0.974241] RPC: Registered named UNIX socket transport module. [ 0.978832] RPC: Registered udp transport module. [ 0.982327] RPC: Registered tcp transport module. [ 0.984359] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.986539] NET: Registered protocol family 44 [ 0.987877] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.989580] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.991323] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.993713] PCI: CLS 0 bytes, default 64 [ 0.995251] Unpacking initramfs... [ 2.399407] debug: unmapping init [mem 0xffff994efcc54000-0xffff994efffbffff] [ 2.403631] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.405963] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.408984] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.932130] Initialise system trusted keyrings [ 2.933626] Key type blacklist registered [ 2.935334] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.944637] zbud: loaded [ 2.947868] *** VALIDATE nfs *** [ 2.949032] *** VALIDATE nfs4 *** [ 2.950619] pstore: using deflate compression [ 2.954467] Platform Keyring initialized [ 3.061048] NET: Registered protocol family 38 [ 3.062803] Key type asymmetric registered [ 3.064123] Asymmetric key parser 'x509' registered [ 3.066051] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.068750] io scheduler mq-deadline registered [ 3.070347] io scheduler kyber registered [ 3.071855] io scheduler bfq registered [ 3.073950] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.077514] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.081449] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.085119] ACPI: Power Button [PWRF] [ 3.091191] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.099477] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.230170] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.240283] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.298362] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.328129] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.360036] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.368037] Non-volatile memory driver v1.3 [ 3.370096] Linux agpgart interface v0.103 [ 3.409965] virtio_blk virtio1: [vda] 149952 512-byte logical blocks (76.8 MB/73.2 MiB) [ 3.413119] vda: detected capacity change from 0 to 76775424 [ 3.434021] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.437008] vdb: detected capacity change from 0 to 1073741824 [ 3.458659] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.462786] vdc: detected capacity change from 0 to 2621440000 [ 3.499925] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.505790] vdd: detected capacity change from 0 to 2621440000 [ 3.525480] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.531284] vde: detected capacity change from 0 to 4294967296 [ 3.554415] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.557321] vdf: detected capacity change from 0 to 4294967296 [ 3.571872] libphy: Fixed MDIO Bus: probed [ 3.618452] usbcore: registered new interface driver usbserial_generic [ 3.621576] usbserial: USB Serial support registered for generic [ 3.623583] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.629353] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.634788] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.637284] mousedev: PS/2 mouse device common for all mice [ 3.642228] rtc_cmos 00:05: RTC can wake from S4 [ 3.644859] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.649025] rtc_cmos 00:05: registered as rtc0 [ 3.652714] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.653389] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.655558] intel_pstate: CPU model not supported [ 3.664728] hid: raw HID events driver (C) Jiri Kosina [ 3.667505] usbcore: registered new interface driver usbhid [ 3.669533] usbhid: USB HID core driver [ 3.671959] drop_monitor: Initializing network drop monitor service [ 3.674611] Initializing XFRM netlink socket [ 3.676934] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.677467] NET: Registered protocol family 10 [ 3.686516] Segment Routing with IPv6 [ 3.687855] NET: Registered protocol family 17 [ 3.691156] mpls_gso: MPLS GSO support [ 3.701873] RAS: Correctable Errors collector initialized. [ 3.703873] AVX version of gcm_enc/dec engaged. [ 3.705962] AES CTR mode by8 optimization enabled [ 3.792482] sched_clock: Marking stable (3792451819, 0)->(4731424292, -938972473) [ 3.796989] registered taskstats version 1 [ 3.799578] Loading compiled-in X.509 certificates [ 3.801567] zswap: loaded using pool lzo/zbud [ 3.826364] Key type big_key registered [ 3.859584] Key type encrypted registered [ 3.861865] ima: No TPM chip found, activating TPM-bypass! [ 3.864912] ima: Allocated hash algorithm: sha1 [ 3.867233] ima: No architecture policies found [ 3.869662] evm: Initialising EVM extended attributes: [ 3.871075] evm: security.selinux [ 3.871928] evm: security.ima [ 3.872697] evm: security.capability [ 3.873649] evm: HMAC attrs: 0x1 [ 3.876253] rtc_cmos 00:05: setting system clock to 2026-09-10 11:12:33 UTC (1789038753) [ 3.887018] debug: unmapping init [mem 0xffffffffaca03000-0xffffffffacbfffff] [ 3.891030] debug: unmapping init [mem 0xffffffffab782000-0xffffffffaba58fff] [ 3.899134] Write protecting the kernel read-only data: 28672k [ 3.906245] debug: unmapping init [mem 0xffffffffa9e03000-0xffffffffa9ffffff] [ 3.908391] debug: unmapping init [mem 0xffffffffaa714000-0xffffffffaa7fffff] [ 3.948567] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.957955] systemd[1]: Detected virtualization kvm. [ 3.960140] systemd[1]: Detected architecture x86-64. [ 3.962409] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.989392] systemd[1]: No hostname configured. [ 3.991472] systemd[1]: Set hostname to . [ 3.993736] random: systemd: uninitialized urandom read (16 bytes read) [ 3.996379] systemd[1]: Initializing machine ID from random generator. [ 4.035912] random: ln: uninitialized urandom read (6 bytes read) [ 4.183686] random: systemd: uninitialized urandom read (16 bytes read) [ 4.186540] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 4.192670] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 4.197833] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Reached target Paths. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. [ OK ] Started Journal Service. Starting dracut pre-udev hook... [ 5.732871] device-mapper: uevent: version 1.0.3 [ 5.736045] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ 6.976940] random: fast init done [ 7.135382] virtio_net virtio0 ens2: renamed from eth0 [ 7.310939] scsi host0: ata_piix [ 7.325248] scsi host1: ata_piix [ 7.326397] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 7.328138] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 11.397970] random: crng init done [ 11.399515] random: 7 urandom warning(s) missed due to ratelimiting [ 13.567495] dracut-initqueue[568]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 15.251374] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 16.834657] printk: systemd: 27 output lines suppressed due to ratelimiting [ 17.171923] SELinux: Disabled at runtime. [ 17.238578] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 17.446281] systemd[1]: Detected virtualization kvm. [ 17.450094] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 18.686406] systemd[1]: initrd-switch-root.service: Succeeded. [ 18.688589] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 18.990038] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 18.993402] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 18.996051] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 19.001455] systemd[1]: Starting Journal Service... Starting Journal Service... [ 19.006696] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target rpc_pipefs.target. Mounting Kernel Debug File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Huge Pages File System... [ OK ] Created slice User and Session Slice. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Create list of required st…ce nodes for the current kernel... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Remount Root and Kernel File Systems... Mounting POSIX Message Queue File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-getty.slice. [ OK ] Reached target Slices. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Listening on Process Core Dump Socket. Starting Apply Kernel Variables... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK [[ 19.380833] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS 0m] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK [[ 19.685985] squashfs: version 4.0 (2009/01/31) Phillip Lougher 0m] Mounted /mnt. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 20.218698] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 20.257443] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 20.521227] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 20.707283] EDAC sbridge: Ver: 1.1.2 [ 22.748897] Key type dns_resolver registered [ 23.072694] NFS: Registering the id_resolver key type [ 23.075239] Key type id_resolver registered [ 23.077456] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg206-server login: [ 63.229440] libcfs: loading out-of-tree module taints kernel. [ 63.365100] Key type ._llcrypt registered [ 63.367023] Key type .llcrypt registered [ 63.552582] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [ 87.227536] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 88.393940] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 88.405953] alg: No test for adler32 (adler32-zlib) [ 90.069240] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 91.235688] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 93.200645] Key type lgssc registered [ 96.016851] Lustre: Echo OBD driver; http://www.lustre.org/ [ 115.317707] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 148.807914] hrtimer: interrupt took 6829630 ns [ 161.351561] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 176.104081] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 176.148691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 177.489544] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 177.561635] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 177.708631] Lustre: lustre-MDT0000: new disk, initializing [ 177.878340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 177.908649] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 183.520746] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 198.669446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 198.865890] Lustre: 6498:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 198.914395] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 198.918430] Lustre: Skipped 1 previous similar message [ 199.030706] Lustre: lustre-MDT0001: new disk, initializing [ 199.103949] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 199.160019] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 199.174327] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 203.788709] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 208.879632] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 217.543301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 217.880129] Lustre: lustre-OST0000: new disk, initializing [ 217.885166] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 217.898448] Lustre: 8439:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 217.981551] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 224.297391] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 224.315783] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 224.374759] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 224.661436] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 239.581696] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 239.838182] Lustre: lustre-OST0001: new disk, initializing [ 239.843165] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 239.850673] Lustre: 9512:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 239.959629] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 246.405323] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 246.420938] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 246.562916] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 248.274037] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 262.483456] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 271.956954] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 279.149653] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing check_logdir /tmp/testlogs/ [ 285.482855] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing yml_node [ 293.429942] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 296.686359] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 299.548446] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 301.401872] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 10 07:17:28 EDT 2026 [ 318.587970] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 320.109249] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 337.376103] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 337.380173] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 337.389505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 338.912507] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 338.912696] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 343.236557] Lustre: server umount lustre-MDT0000 complete [ 347.891211] LustreError: 6492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789039097 with bad export cookie 4224545311634815244 [ 347.898117] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 347.908095] LustreError: 6492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 349.153172] LustreError: 6506:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 349.161141] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 349.161507] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 349.161512] Lustre: Skipped 3 previous similar messages [ 349.180598] LustreError: 6506:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 349.215251] Lustre: Skipped 3 previous similar messages [ 352.161319] LustreError: 6506:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 352.162443] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 352.186056] LustreError: 6506:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 352.200089] Lustre: Skipped 2 previous similar messages [ 354.446166] Lustre: server umount lustre-MDT0001 complete [ 358.841847] Lustre: server umount lustre-OST0000 complete [ 363.495640] Lustre: server umount lustre-OST0001 complete [ 371.964500] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [ 383.392294] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 427.009829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 427.378345] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 427.418355] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 427.511925] Lustre: lustre-MDT0000: new disk, initializing [ 427.565263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 427.579223] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 433.884885] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 448.890994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 449.154620] Lustre: 15071:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 449.282424] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 449.291314] Lustre: Skipped 1 previous similar message [ 449.475782] Lustre: lustre-MDT0001: new disk, initializing [ 449.687112] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 449.816981] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 449.825904] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 455.453716] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 470.230353] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 477.346963] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 484.870105] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 485.151682] Lustre: lustre-OST0000: new disk, initializing [ 485.162617] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 485.173100] Lustre: 17042:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 485.263534] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 487.103804] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 487.119273] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 487.242094] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 492.304930] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 504.732493] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 511.168584] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 511.546670] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 517.788564] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 518.027651] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 523.237927] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 523.245741] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 523.259565] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 523.287335] Lustre: Skipped 1 previous similar message [ 527.257374] Lustre: server umount lustre-OST0000 complete [ 533.474505] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 533.482498] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 533.509802] Lustre: Skipped 1 previous similar message [ 533.528776] Lustre: Skipped 2 previous similar messages [ 538.601667] LustreError: 15077:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 538.628109] LustreError: 15077:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 538.686704] Lustre: server umount lustre-MDT0000 complete [ 542.707303] LustreError: 15079:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 542.747624] LustreError: 15079:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 543.823138] LustreError: 15062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789039293 with bad export cookie 4224545311634816777 [ 543.828057] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 543.835370] LustreError: 15062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 544.526821] Lustre: server umount lustre-MDT0001 complete [ 552.915606] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 07:21:39 (1789039299) [ 583.905395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 584.871987] LustreError: 19919:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 585.003891] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 589.558992] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 594.400960] LustreError: 19920:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 594.426823] LustreError: 19920:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 599.676281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 604.469802] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 612.078565] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 619.623619] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 621.829981] Lustre: Failing over lustre-MDT0000 [ 622.029869] Lustre: server umount lustre-MDT0000 complete [ 624.931740] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 624.946279] LustreError: 20648:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 624.958381] LustreError: 20648:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 640.479177] Lustre: 3631:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789039374/real 1789039374] req@ffff994f771a1f80 x1875943190861824/t0(0) o400->MGC192.168.202.106@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789039390 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 640.534899] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 643.890116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 652.188532] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 652.200868] Lustre: Skipped 1 previous similar message [ 652.293272] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 656.866426] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 657.380379] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 657.387883] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 657.420178] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 669.090166] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 674.664786] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 674.706771] Lustre: Skipped 1 previous similar message [ 674.728334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 674.741745] Lustre: Skipped 1 previous similar message [ 681.112257] Lustre: server umount lustre-MDT0000 complete [ 685.024050] LustreError: 19919:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 685.070542] LustreError: 19919:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 15 previous similar messages [ 686.094209] LustreError: 19899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789039435 with bad export cookie 4224545311634818191 [ 686.110562] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 686.111052] LustreError: 19899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 686.616555] Lustre: server umount lustre-MDT0001 complete [ 695.994119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 702.541852] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 712.459454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 718.123780] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 725.986988] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 732.287231] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 734.774168] Lustre: Failing over lustre-MDT0000 [ 737.089388] Lustre: server umount lustre-MDT0000 complete [ 737.766759] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 737.778616] Lustre: Skipped 1 previous similar message [ 749.024961] LustreError: 23258:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 749.054415] LustreError: 23258:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 754.143234] Lustre: 3628:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789039487/real 1789039487] req@ffff994e43843480 x1875943190892544/t0(0) o400->MGC192.168.202.106@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789039503 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 754.204207] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 758.936094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 764.770827] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 770.018879] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 770.043368] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 770.050382] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 770.065760] Lustre: Skipped 1 previous similar message [ 770.090804] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 780.350845] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 785.378985] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 785.397774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 785.400093] Lustre: Skipped 2 previous similar messages [ 791.099522] Lustre: server umount lustre-MDT0000 complete [ 794.573835] LustreError: 23243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789039544 with bad export cookie 4224545311634819129 [ 794.586763] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 794.599839] LustreError: 23243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 795.057415] Lustre: server umount lustre-MDT0001 complete [ 805.396547] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 07:25:52 (1789039552) [ 826.191371] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [ 837.270540] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 884.985495] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 898.574341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 898.854771] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 898.901742] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 899.017972] Lustre: lustre-MDT0000: new disk, initializing [ 899.129871] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 899.135536] Lustre: Skipped 3 previous similar messages [ 899.153853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 904.859139] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 917.270795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 917.528881] Lustre: 29638:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 918.189728] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 924.109682] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 935.666087] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 945.308369] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 953.901349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 954.561751] Lustre: 31609:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 956.677682] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 956.782850] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 956.948506] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 956.971194] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 963.125281] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 973.647980] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 980.208285] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 1002.976789] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1002.998610] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1003.008234] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1003.025607] Lustre: Skipped 4 previous similar messages [ 1003.040221] Lustre: Skipped 3 previous similar messages [ 1017.311149] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1017.774327] Lustre: server umount lustre-OST0000 complete [ 1036.255551] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1036.523155] Lustre: server umount lustre-MDT0000 complete [ 1038.816888] LustreError: 30304:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1038.827729] LustreError: 30304:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [ 1040.788785] LustreError: 29629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789039790 with bad export cookie 4224545311634819535 [ 1040.790222] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1040.799691] LustreError: 29629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1041.279480] Lustre: server umount lustre-MDT0001 complete [ 1053.593323] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 1057.003895] Key type lgssc unregistered [ 1057.543764] LNet: 33589:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1057.551300] LNetError: 33589:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1057.566151] LNet: Removed LNI 192.168.202.106@tcp [ 1058.965964] Key type .llcrypt unregistered [ 1058.967814] Key type ._llcrypt unregistered [ 1085.998371] Key type ._llcrypt registered [ 1086.000678] Key type .llcrypt registered [ 1086.106892] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 1087.588669] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1087.675741] alg: No test for adler32 (adler32-zlib) [ 1088.846916] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 1089.084922] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 1090.775402] Key type lgssc registered [ 1091.955420] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1101.060911] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1101.083971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1102.529937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1108.215590] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 1113.057084] LustreError: 35001:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1113.065097] LustreError: 35001:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1117.904690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1118.179375] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1118.402874] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1124.391647] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 1134.194984] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1140.965572] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1147.994454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1148.435404] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1153.570579] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1155.598239] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 1162.743631] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1163.666732] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1178.930632] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 07:32:06 (1789039926) [ 1181.086372] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1183.036810] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1191.050638] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 07:32:18 (1789039938) [ 1201.301159] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 07:32:27 (1789039947) [ 1202.440866] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1212.564606] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1212.585105] Lustre: Skipped 2 previous similar messages [ 1214.735384] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 07:32:41 (1789039961) [ 1216.788625] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1216.794840] Lustre: Skipped 2 previous similar messages [ 1225.482333] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1225.489288] Lustre: Skipped 4 previous similar messages [ 1240.139220] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 07:33:07 (1789039987) [ 1242.927670] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1247.005605] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1256.873727] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 07:33:24 (1789040004) [ 1258.894711] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1258.897309] Lustre: Skipped 6 previous similar messages [ 1273.644439] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 07:33:40 (1789040020) [ 1291.748626] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1291.752890] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1291.768073] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1295.841928] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1295.848205] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1295.859740] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1297.568282] Lustre: server umount lustre-OST0000 complete [ 1302.496651] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1302.517247] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1302.540917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1302.547766] Lustre: Skipped 1 previous similar message [ 1306.085582] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1308.521838] Lustre: server umount lustre-MDT0000 complete [ 1311.201170] LustreError: 35001:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1311.231534] LustreError: 35001:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1314.111734] LustreError: 36844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789040063 with bad export cookie 17006331933306276376 [ 1314.113351] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1314.130827] LustreError: 36844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1314.758945] Lustre: server umount lustre-MDT0001 complete [ 1329.267654] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 1331.903582] Key type lgssc unregistered [ 1332.242661] LNet: 42863:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1332.256934] LNetError: 42863:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1332.271654] LNet: Removed LNI 192.168.202.106@tcp [ 1333.205350] Key type .llcrypt unregistered [ 1333.208113] Key type ._llcrypt unregistered [ 1350.721158] Key type ._llcrypt registered [ 1350.722996] Key type .llcrypt registered [ 1350.831896] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [ 1368.315251] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 1369.268222] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1369.354974] alg: No test for adler32 (adler32-zlib) [ 1370.472448] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 1370.667210] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 1372.311446] Key type lgssc registered [ 1373.621062] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1411.264354] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1411.323098] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1412.547363] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1412.580255] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1412.687496] Lustre: lustre-MDT0000: new disk, initializing [ 1412.805139] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1412.828206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1418.048744] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 1429.013914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1429.116388] Lustre: 46010:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1429.163759] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1429.167738] Lustre: Skipped 1 previous similar message [ 1429.230442] Lustre: lustre-MDT0001: new disk, initializing [ 1429.299487] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1429.341234] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1429.353354] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1434.213301] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 1443.096282] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1448.959336] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1455.295645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1455.641681] Lustre: lustre-OST0000: new disk, initializing [ 1455.645237] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1455.651692] Lustre: 47978:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1455.733552] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1462.284847] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 1462.865048] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1462.872022] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1462.984884] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1471.878798] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1477.104252] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1477.296385] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1482.117367] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1482.298768] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1488.872602] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1488.883415] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1488.903914] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1488.917883] Lustre: Skipped 1 previous similar message [ 1490.691593] Lustre: server umount lustre-OST0000 complete [ 1496.032593] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1496.052061] LustreError: Skipped 1 previous similar message [ 1496.061048] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1496.075920] Lustre: Skipped 1 previous similar message [ 1496.090835] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1499.108248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1499.109430] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1504.224557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1504.227880] Lustre: Skipped 2 previous similar messages [ 1509.345156] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1509.347155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1509.365643] Lustre: Skipped 1 previous similar message [ 1509.594919] Lustre: server umount lustre-MDT0000 complete [ 1514.018512] LustreError: 46005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789040263 with bad export cookie 16839436118204407363 [ 1514.032101] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1514.034384] LustreError: 46005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1514.687775] Lustre: server umount lustre-MDT0001 complete [ 1525.894837] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 07:37:52 (1789040272) [ 1533.060896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1533.792227] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1538.587413] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 1539.041357] LustreError: 50442:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1539.059671] LustreError: 50442:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1544.169022] LustreError: 50443:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1547.425123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1547.811429] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1552.636156] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 1561.266122] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1567.119652] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1574.998344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1575.391718] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1585.503628] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 1595.717245] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1601.155540] LustreError: 52971:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1602.282138] Lustre: Modifying parameter general.timeout=1 in log params [ 1604.001938] Lustre: Modifying parameter general.timeout=3 in log params [ 1604.009587] Lustre: Skipped 1 previous similar message [ 1606.115682] Lustre: Modifying parameter general.timeout=5 in log params [ 1606.122143] Lustre: Skipped 1 previous similar message [ 1610.323617] Lustre: Modifying parameter general.timeout=9 in log params [ 1610.326546] Lustre: Skipped 3 previous similar messages [ 1618.607589] Lustre: Modifying parameter general.timeout=17 in log params [ 1618.610054] Lustre: Skipped 7 previous similar messages [ 1635.253970] Lustre: Modifying parameter general.timeout=33 in log params [ 1635.261552] Lustre: Skipped 15 previous similar messages [ 1668.244292] Lustre: Modifying parameter general.timeout=66 in log params [ 1668.248740] Lustre: Skipped 32 previous similar messages [ 1734.148696] Lustre: Disabling parameter general.timeout= in log params [ 1734.154201] Lustre: Skipped 63 previous similar messages [ 1738.038962] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 07:41:25 (1789040485) [ 1741.059847] Lustre: 59923:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1742.972433] Lustre: 60021:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1746.982753] Lustre: 60218:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1746.998177] Lustre: 60218:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1751.009775] Lustre: 60431:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1751.027237] Lustre: 60431:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1777.501678] Lustre: 61790:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1777.516240] LustreError: 61790:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1782.367651] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 07:42:09 (1789040529) [ 1798.625863] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1798.641281] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1798.665338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1802.996785] Lustre: server umount lustre-MDT0000 complete [ 1806.312704] LustreError: 50443:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1806.332261] LustreError: 50443:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1807.015554] LustreError: 57383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789040556 with bad export cookie 16839436118204408245 [ 1807.022489] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1807.025047] LustreError: 57383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1807.287602] Lustre: server umount lustre-MDT0001 complete [ 1822.292695] Lustre: server umount lustre-OST0000 complete [ 1855.528374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1856.013663] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1856.323937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1861.445761] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 1870.455711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1870.536419] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1870.582244] Lustre: 64323:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1875.540785] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 1883.952784] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1890.139231] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1898.006363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1898.203258] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1898.703496] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1898.714235] Lustre: Skipped 1 previous similar message [ 1906.338552] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 1915.613726] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1917.398910] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1917.407844] Lustre: Skipped 23 previous similar messages [ 1917.456420] Lustre: 66888:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1928.690523] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1928.693104] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1928.701666] LustreError: Skipped 1 previous similar message [ 1928.702712] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1928.707948] Lustre: Skipped 3 previous similar messages [ 1928.724850] Lustre: Skipped 4 previous similar messages [ 1931.242121] Lustre: server umount lustre-OST0000 complete [ 1937.376739] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1937.382609] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1941.381423] Lustre: server umount lustre-MDT0000 complete [ 1944.033878] LustreError: 64330:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1944.066802] LustreError: 64330:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1945.134446] LustreError: 64313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789040694 with bad export cookie 16839436118204411297 [ 1945.142001] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1945.144226] LustreError: 64313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1945.458275] Lustre: server umount lustre-MDT0001 complete [ 1958.761479] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 1961.753316] Key type lgssc unregistered [ 1962.112796] LNet: 68097:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1962.119105] LNetError: 68097:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1963.180367] LNet: Removed LNI 192.168.202.106@tcp [ 1964.000363] Key type .llcrypt unregistered [ 1964.006920] Key type ._llcrypt unregistered [ 1989.155922] Key type ._llcrypt registered [ 1989.159633] Key type .llcrypt registered [ 1989.304549] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 1990.784348] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1990.823670] alg: No test for adler32 (adler32-zlib) [ 1991.963874] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 1992.208863] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 1993.911221] Key type lgssc registered [ 1994.939554] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2002.660278] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2002.692032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2004.077566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2009.320571] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 2013.669715] LustreError: 69511:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2013.684046] LustreError: 69511:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2018.671486] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2018.785134] LustreError: 69510:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2019.671721] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2024.530842] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 2033.027467] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2039.714354] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2048.308834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2048.486084] LustreError: 71336:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2048.969859] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2055.950083] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 2064.324253] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2074.753460] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2085.868247] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2085.874066] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2085.874705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2085.914817] Lustre: Skipped 2 previous similar messages [ 2090.977359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2090.986650] Lustre: Skipped 4 previous similar messages [ 2092.238090] Lustre: server umount lustre-MDT0000 complete [ 2096.103307] LustreError: 69510:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2096.124058] LustreError: 69510:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2096.286695] LustreError: 72194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789040845 with bad export cookie 10644333335661920616 [ 2096.306228] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2096.306466] LustreError: 72194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2096.813962] Lustre: server umount lustre-MDT0001 complete [ 2111.397553] Lustre: server umount lustre-OST0000 complete [ 2142.848134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2143.182291] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2143.204195] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2143.391725] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2148.592648] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 2159.575677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2159.678994] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2159.705772] Lustre: 74383:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2160.177513] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2165.077622] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 2174.559099] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2181.939845] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2191.534034] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2191.885213] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2192.464819] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2199.715460] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 2209.106714] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2210.880904] Lustre: 76957:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2213.442192] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2221.024936] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2221.034816] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2221.060481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2225.752080] Lustre: server umount lustre-OST0000 complete [ 2231.777063] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2231.781775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2231.807795] Lustre: Skipped 2 previous similar messages [ 2231.821081] Lustre: Skipped 2 previous similar messages [ 2235.581167] Lustre: server umount lustre-MDT0000 complete [ 2236.901536] LustreError: 77121:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2236.921936] LustreError: 77121:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2240.661579] LustreError: 75127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789040990 with bad export cookie 10644333335661921470 [ 2240.669195] LustreError: 75127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2240.673969] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2241.518923] Lustre: server umount lustre-MDT0001 complete [ 2255.667202] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 2258.781195] Key type lgssc unregistered [ 2259.090771] LNet: 78167:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2259.100115] LNetError: 78167:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2260.133880] LNet: Removed LNI 192.168.202.106@tcp [ 2261.032490] Key type .llcrypt unregistered [ 2261.035056] Key type ._llcrypt unregistered [ 2291.162309] Key type ._llcrypt registered [ 2291.164500] Key type .llcrypt registered [ 2291.283707] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 2292.359463] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2292.411848] alg: No test for adler32 (adler32-zlib) [ 2293.495549] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 2293.688838] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 2295.328825] Key type lgssc registered [ 2296.407138] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2304.424509] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2304.449924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2305.989554] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2311.694745] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 2316.271798] LustreError: 79580:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2316.297664] LustreError: 79580:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2321.380865] LustreError: 79579:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2322.351132] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2323.384046] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2329.800687] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 2338.795721] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2345.820294] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2353.601334] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2354.213645] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2362.074307] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 2370.276946] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2379.868363] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2380.815214] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2382.414563] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2382.423359] Lustre: Skipped 1 previous similar message [ 2385.215968] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2385.221829] Lustre: Skipped 2 previous similar messages [ 2389.810815] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2389.813785] Lustre: Skipped 3 previous similar messages [ 2398.661249] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2398.665178] Lustre: Skipped 9 previous similar messages [ 2415.546795] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2415.549206] Lustre: Skipped 16 previous similar messages [ 2448.356223] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2448.359354] Lustre: Skipped 34 previous similar messages [ 2486.715835] Lustre: 87682:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2489.349321] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 07:53:56 (1789041236) [ 2491.331448] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2493.318304] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 07:54:00 (1789041240) [ 2498.528850] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2498.533247] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2498.536024] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2498.551590] Lustre: Skipped 1 previous similar message [ 2503.669311] LustreError: 82082:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2503.797841] Lustre: server umount lustre-OST0000 complete [ 2508.771555] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2508.783166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2508.799634] Lustre: Skipped 1 previous similar message [ 2508.813251] Lustre: Skipped 2 previous similar messages [ 2513.889098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2513.906170] Lustre: Skipped 1 previous similar message [ 2514.269376] Lustre: server umount lustre-MDT0000 complete [ 2518.130521] LustreError: 79559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789041267 with bad export cookie 8606131524170217385 [ 2518.132625] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2518.138607] LustreError: 79559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2518.766820] Lustre: server umount lustre-MDT0001 complete [ 2532.634266] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 2536.203623] Key type lgssc unregistered [ 2536.491813] LNet: 88928:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2536.496922] LNetError: 88928:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2536.512213] LNet: Removed LNI 192.168.202.106@tcp [ 2537.378723] Key type .llcrypt unregistered [ 2537.381331] Key type ._llcrypt unregistered [ 2542.287274] Key type ._llcrypt registered [ 2542.289592] Key type .llcrypt registered [ 2542.423046] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2548.041636] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules [ 2549.610701] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2549.754683] alg: No test for adler32 (adler32-zlib) [ 2550.836794] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 2551.199745] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 2552.903505] Key type lgssc registered [ 2554.784824] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2556.474622] LustreError: 90171:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2575.896205] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 2579.338091] LustreError: 90171:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2585.622404] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2585.650961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2587.316039] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2592.534802] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 2596.835379] LustreError: 91179:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2596.857504] LustreError: 91179:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2601.955943] LustreError: 91178:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2602.893526] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 07:55:49 (1789041349) [ 2607.088336] LustreError: 91179:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2607.922500] LustreError: 91210:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2607.945083] LustreError: 91210:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2607.973814] LustreError: 91210:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 2608.400328] Lustre: server umount lustre-MDT0000 complete [ 2625.115685] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 2628.704457] Key type lgssc unregistered [ 2629.066808] LNet: 92704:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2629.080550] LNetError: 92704:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2630.117764] LNet: Removed LNI 192.168.202.106@tcp [ 2631.128229] Key type .llcrypt unregistered [ 2631.130283] Key type ._llcrypt unregistered [ 2650.175847] Key type ._llcrypt registered [ 2650.180935] Key type .llcrypt registered [ 2650.313874] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 2651.431785] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2651.650787] alg: No test for adler32 (adler32-zlib) [ 2652.728297] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 2652.964271] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 2654.615321] Key type lgssc registered [ 2655.748635] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2664.124523] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2664.155234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2665.649153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2670.475488] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 2676.197131] LustreError: 93916:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2676.221854] LustreError: 93916:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2679.717573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2680.288157] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2685.032464] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 2692.367771] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2698.385798] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2705.375143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2705.796668] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2712.021292] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 2719.262584] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2771.966578] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 07:58:39 (1789041519) [ 2773.867225] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2775.900287] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 07:58:42 (1789041522) [ 2781.669580] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2781.676397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2781.684418] Lustre: Skipped 2 previous similar messages [ 2781.704206] Lustre: Skipped 2 previous similar messages [ 2786.720752] Lustre: server umount lustre-MDT0000 complete [ 2786.792157] LustreError: 93916:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2786.820939] LustreError: 93916:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2791.718100] LustreError: 93897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789041541 with bad export cookie 3219728266553265262 [ 2791.731194] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2791.732639] LustreError: 93897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2791.904320] LustreError: 94924:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2791.909227] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2791.942957] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2792.298943] Lustre: server umount lustre-MDT0001 complete [ 2800.017461] Lustre: server umount lustre-OST0000 complete [ 2813.698315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2814.141820] LustreError: 98214:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2814.260433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2819.537939] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 2819.554916] LustreError: 98215:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2828.781094] LustreError: 98214:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2828.806883] LustreError: 98214:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2829.298779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2830.025027] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2833.847196] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 2840.831933] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2847.246128] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2862.993279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2863.266989] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2863.274655] LustreError: 98203:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2863.297401] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2863.303826] LustreError: 100298:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 2863.308556] LustreError: 100298:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 2863.312319] LustreError: 100298:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2863.425789] Lustre: server umount lustre-OST0000 complete [ 2863.428715] LustreError: 100298:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2875.324679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2875.383843] LustreError: 100773:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2875.493373] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2875.503573] LustreError: 98204:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2875.527192] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2875.535380] LustreError: 100766:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 2875.546919] LustreError: 100766:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 2875.559035] LustreError: 100766:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2875.653660] Lustre: server umount lustre-OST0000 complete [ 2875.659674] LustreError: 100766:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2885.749678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2886.018578] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2886.024800] Lustre: Found index 0 for lustre-OST0000, updating log [ 2886.034542] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2886.215440] Lustre: lustre-OST0000: new disk, initializing [ 2886.224465] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2886.237402] Lustre: 101302:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2886.661519] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2893.063816] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 2896.791035] Lustre: Failing over lustre-OST0000 [ 2896.984197] Lustre: server umount lustre-OST0000 complete [ 2901.473196] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2901.490483] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2901.522882] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2906.084156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2906.092994] Lustre: Skipped 2 previous similar messages [ 2906.846462] Lustre: server umount lustre-MDT0000 complete [ 2910.830881] LustreError: 98195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789041660 with bad export cookie 3219728266553268006 [ 2910.835765] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2910.858362] LustreError: 98195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2911.226659] Lustre: server umount lustre-MDT0001 complete [ 2923.072185] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 08:01:09 (1789041669) [ 2945.069477] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 2958.307570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2958.864652] LustreError: 103660:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2959.003447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2964.707880] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 2975.285317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2982.053566] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 2985.151626] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2992.570620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2993.506861] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2993.515804] Lustre: Skipped 1 previous similar message [ 3000.969538] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3009.653214] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3009.728418] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3009.836109] Lustre: lustre-OST0001: new disk, initializing [ 3009.848399] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3009.862181] Lustre: 105980:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3015.813378] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3015.837218] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3015.981720] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3017.658416] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3026.154567] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3033.853397] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3041.735080] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.6@tcp (stopping) [ 3041.771121] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3041.806816] Lustre: Skipped 2 previous similar messages [ 3044.625743] Lustre: server umount lustre-MDT0001 complete [ 3046.812373] LustreError: 103656:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3046.832974] LustreError: 103656:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 3055.415163] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3055.539046] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3055.542465] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3055.571860] Lustre: 103650:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3056.923022] LustreError: 107514:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3056.941266] LustreError: 107514:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3056.958951] LustreError: 103652:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3057.076475] LustreError: 107521:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.106@tcp: cfg command failed: rc = -17 [ 3057.084571] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3057.084571] [ 3061.262495] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3061.718317] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3073.634313] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 08:03:41 (1789041821) [ 3093.053458] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 3113.859586] Lustre: 109594:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3139.668672] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3144.704305] Lustre: 110730:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3250.146620] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3250.155841] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3250.174079] Lustre: Skipped 1 previous similar message [ 3250.183706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3250.192745] Lustre: Skipped 3 previous similar messages [ 3252.450780] Lustre: server umount lustre-MDT0000 complete [ 3255.777482] LustreError: 110918:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3255.801095] LustreError: 110918:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 13 previous similar messages [ 3256.612666] LustreError: 103641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789042006 with bad export cookie 3219728266553268993 [ 3256.615404] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3256.623520] LustreError: 103641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3257.027637] Lustre: server umount lustre-MDT0001 complete [ 3271.705533] Lustre: server umount lustre-OST0000 complete [ 3276.454327] Lustre: 93229:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789042010/real 1789042010] req@ffff994e48874a80 x1875945877518336/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789042026 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3276.483494] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3276.510049] Lustre: Skipped 3 previous similar messages [ 3277.734424] Lustre: server umount lustre-OST0001 complete [ 3285.353141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3298.612979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3307.822634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3319.015724] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3344.582599] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 3354.986288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3355.000481] Lustre: lustre-MDT0000: reset Object Index mappings [ 3355.453323] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3355.459398] Lustre: Skipped 2 previous similar messages [ 3359.181117] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3369.222141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3369.248990] Lustre: lustre-MDT0001: reset Object Index mappings [ 3375.333731] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3378.530156] Lustre: 115873:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3386.353867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3393.205556] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3394.497538] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 3399.667247] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 3403.922875] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3410.454586] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3410.465091] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3412.628385] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3421.885935] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3425.821258] Lustre: 117751:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3482.757429] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 08:10:30 (1789042230) [ 3508.192076] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3508.195935] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3508.205525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3508.208830] Lustre: Skipped 3 previous similar messages [ 3514.487353] Lustre: server umount lustre-MDT0000 complete [ 3517.931984] LustreError: 114724:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3517.967073] LustreError: 114724:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [ 3518.873615] LustreError: 114705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789042268 with bad export cookie 3219728266553315865 [ 3518.883134] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3518.887640] LustreError: 114705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3533.279887] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3533.571290] Lustre: server umount lustre-MDT0001 complete [ 3552.223249] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3552.348100] Lustre: server umount lustre-OST0000 complete [ 3562.580690] Lustre: server umount lustre-OST0001 complete [ 3569.174976] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [ 3580.685738] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 3625.461308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3625.705179] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3625.726814] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3625.789759] Lustre: lustre-MDT0000: new disk, initializing [ 3625.866193] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3625.871509] Lustre: Skipped 3 previous similar messages [ 3625.883603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3630.345705] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3638.738662] Lustre: server umount lustre-MDT0000 complete [ 3650.651069] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3651.037676] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3651.070831] Lustre: 122041:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3651.093903] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3655.972975] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3661.687620] Lustre: server umount lustre-MDT0000 complete [ 3672.932620] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 08:13:39 (1789042419) [ 3674.689074] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3676.520726] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 08:13:43 (1789042423) [ 3695.760834] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 3706.744589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3711.772828] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3720.866396] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3721.025925] Lustre: 124090:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3721.080209] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3721.097661] Lustre: Skipped 1 previous similar message [ 3721.210279] Lustre: lustre-MDT0001: new disk, initializing [ 3721.350183] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3721.365802] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3725.992149] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3730.921417] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3737.504888] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3737.760941] Lustre: lustre-OST0000: new disk, initializing [ 3737.769632] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3737.780780] Lustre: 125667:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3739.822214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3739.836965] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3739.964490] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3744.349457] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3756.260206] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3756.492317] Lustre: lustre-OST0001: new disk, initializing [ 3756.499524] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3756.510653] Lustre: 126539:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3758.226753] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3758.255614] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3758.373879] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3764.091409] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3776.212072] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3781.286296] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4188.119931] Lustre: 132952:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4188.137288] Lustre: 132952:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4188.143321] Lustre: 132952:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4188.160881] Lustre: 132952:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4188.169241] Lustre: 132952:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4188.174821] Lustre: 132952:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4462.395361] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 08:26:48 (1789043208) [ 4478.438369] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4478.442656] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4478.451949] Lustre: Skipped 8 previous similar messages [ 4478.469674] Lustre: Skipped 13 previous similar messages [ 4483.557063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4483.569552] Lustre: Skipped 3 previous similar messages [ 4484.304262] Lustre: server umount lustre-MDT0000 complete [ 4487.967261] LustreError: 124084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789043237 with bad export cookie 3219728266553385081 [ 4487.970499] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4487.987245] LustreError: 124084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4488.677641] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4488.680844] LustreError: 124099:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4488.698838] LustreError: 124099:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 4493.803480] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4493.808567] Lustre: Skipped 3 previous similar messages [ 4494.650262] Lustre: server umount lustre-MDT0001 complete [ 4504.880838] Lustre: server umount lustre-OST0000 complete [ 4522.975893] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4523.393617] Lustre: server umount lustre-OST0001 complete [ 4531.540697] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [ 4543.356920] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 4591.360909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4591.675295] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4591.717802] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4591.847971] Lustre: lustre-MDT0000: new disk, initializing [ 4591.950128] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4591.957646] Lustre: Skipped 5 previous similar messages [ 4592.000245] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4597.417329] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 4609.129814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4609.379373] Lustre: 136004:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4609.435445] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4609.443056] Lustre: Skipped 1 previous similar message [ 4609.496346] Lustre: lustre-MDT0001: new disk, initializing [ 4609.614730] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4609.628752] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4615.057399] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 4626.358745] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4633.989411] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4645.234869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4645.697212] Lustre: lustre-OST0000: new disk, initializing [ 4645.700442] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4645.707377] Lustre: 137976:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4645.795075] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4645.799314] Lustre: Skipped 1 previous similar message [ 4646.794715] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4646.809391] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4646.974186] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4652.609426] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 4663.783374] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4675.042478] Lustre: lustre-MDD0000: changelog on [ 4678.211154] Lustre: lustre-MDD0001: changelog on [ 6655.498258] Lustre: lustre-MDD0001: changelog off [ 6657.301774] Lustre: 136011:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6658.230876] Lustre: lustre-MDD0000: changelog off [ 6668.285573] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6668.300256] Lustre: Skipped 2 previous similar messages [ 6668.300960] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6668.307416] Lustre: Skipped 1 previous similar message [ 6671.241362] Lustre: server umount lustre-OST0000 complete [ 6677.991223] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6677.999505] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6678.019823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6681.991442] Lustre: server umount lustre-MDT0000 complete [ 6683.615841] LustreError: 136872:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6683.625723] LustreError: 136872:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 6685.675135] LustreError: 146187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789045435 with bad export cookie 3219728266554162095 [ 6685.681794] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6685.695545] LustreError: 146187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6686.055732] Lustre: server umount lustre-MDT0001 complete [ 6697.079286] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 6699.303514] Key type lgssc unregistered [ 6699.625520] LNet: 147140:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6699.633707] LNetError: 147140:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6699.657477] LNet: Removed LNI 192.168.202.106@tcp [ 6700.729195] Key type .llcrypt unregistered [ 6700.730765] Key type ._llcrypt unregistered [ 6721.426763] Key type ._llcrypt registered [ 6721.430938] Key type .llcrypt registered [ 6721.649857] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [ 6739.480593] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 6740.771105] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6740.817901] alg: No test for adler32 (adler32-zlib) [ 6741.955042] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 6742.144388] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 6743.783348] Key type lgssc registered [ 6744.940288] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6779.793211] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 6779.847899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6781.171561] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6781.220899] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6781.353700] Lustre: lustre-MDT0000: new disk, initializing [ 6781.489904] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6781.516241] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6786.117502] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6796.684507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6796.856909] Lustre: 150348:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6796.915476] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6796.919146] Lustre: Skipped 1 previous similar message [ 6796.997249] Lustre: lustre-MDT0001: new disk, initializing [ 6797.131625] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6797.173760] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6797.184764] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6802.194851] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6812.156767] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6817.599737] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6824.399767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6824.617129] Lustre: lustre-OST0000: new disk, initializing [ 6824.620750] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6824.627743] Lustre: 152318:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6824.717531] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6830.515886] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6833.786199] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6833.810256] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6833.913437] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6840.702985] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6846.472723] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_module obdecho/obdecho [ 6848.413802] LustreError: 153320:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6848.422600] LustreError: 153320:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6850.085521] LustreError: 153416:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6863.847739] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6863.856852] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6863.880194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6864.870262] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6864.874571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6864.878189] Lustre: Skipped 1 previous similar message [ 6864.886027] Lustre: Skipped 2 previous similar messages [ 6866.926344] Lustre: server umount lustre-MDT0000 complete [ 6871.195037] LustreError: 150340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789045620 with bad export cookie 4020457121176163975 [ 6871.200534] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6871.208767] LustreError: 150340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6871.636969] Lustre: server umount lustre-MDT0001 complete [ 6885.582197] Lustre: server umount lustre-OST0000 complete [ 6907.443839] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [ 6918.301472] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 6960.302733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6960.664864] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6960.707693] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6960.875650] Lustre: lustre-MDT0000: new disk, initializing [ 6961.003945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6961.053448] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6966.919305] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6979.758086] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6979.898648] Lustre: 157134:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6979.948801] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6979.956151] Lustre: Skipped 1 previous similar message [ 6980.031910] Lustre: lustre-MDT0001: new disk, initializing [ 6980.133193] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6980.179745] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6980.198523] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6984.589873] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6994.914235] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7002.653509] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7010.867588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7011.186769] Lustre: lustre-OST0000: new disk, initializing [ 7011.192392] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7011.203973] Lustre: 159105:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7011.295582] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7018.801526] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7020.087240] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7020.097723] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7020.233143] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7029.533905] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7034.754435] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7035.145121] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7040.783955] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7041.118654] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7046.112783] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7046.116239] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7046.119919] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7046.128120] LustreError: Skipped 1 previous similar message [ 7046.155024] Lustre: Skipped 1 previous similar message [ 7049.650694] Lustre: server umount lustre-OST0000 complete [ 7056.356164] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7056.368983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7056.379596] Lustre: Skipped 1 previous similar message [ 7056.384768] Lustre: Skipped 1 previous similar message [ 7060.575224] Lustre: server umount lustre-MDT0000 complete [ 7061.474601] LustreError: 158460:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7061.496337] LustreError: 158460:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 7065.428094] LustreError: 157125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789045815 with bad export cookie 4020457121176164934 [ 7065.433420] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7065.443836] LustreError: 157125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7065.752453] Lustre: server umount lustre-MDT0001 complete [ 7069.959865] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 09:10:16 (1789045816) [ 7072.176649] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7074.420655] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 09:10:21 (1789045821) [ 7083.612359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7084.184915] LustreError: 161517:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7084.250030] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7088.669711] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7089.645133] LustreError: 161518:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7093.728400] LustreError: 161517:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7098.222948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7104.277850] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7114.812515] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7123.307783] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7131.193827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7131.550808] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7131.558381] Lustre: Skipped 1 previous similar message [ 7138.553358] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7146.526537] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7164.402356] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7164.406401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7164.441460] Lustre: Skipped 2 previous similar messages [ 7164.464303] Lustre: Skipped 3 previous similar messages [ 7169.385571] Lustre: server umount lustre-MDT0000 complete [ 7169.524076] LustreError: 161518:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7169.548920] LustreError: 161518:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7173.705910] LustreError: 161500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789045923 with bad export cookie 4020457121176165816 [ 7173.728373] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7174.625924] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7174.661901] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7174.680078] Lustre: Skipped 1 previous similar message [ 7178.160191] LustreError: 161512:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7178.201625] LustreError: 161512:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7180.258980] Lustre: server umount lustre-MDT0001 complete [ 7187.118229] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7200.022507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7200.396769] LustreError: 165382:0:(llog.c:1655:llog_backup()) MGC192.168.202.106@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7200.403110] Lustre: 165382:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.106@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7204.320390] LustreError: 165387:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7204.332539] LustreError: 165387:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7239.138633] LustreError: 165387:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7239.163524] LustreError: 165387:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 25 previous similar messages [ 7261.151898] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7261.161136] Lustre: 165382:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.106@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7302.158145] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7305.114670] LustreError: 165389:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7305.127086] LustreError: 165389:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 44 previous similar messages [ 7307.253033] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7307.290211] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7317.471652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7324.071951] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7324.138410] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7333.810653] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7335.906244] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7341.959852] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7343.713877] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7365.111588] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7365.113331] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7365.135933] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7365.135942] Lustre: Skipped 2 previous similar messages [ 7366.616523] Lustre: server umount lustre-MDT0000 complete [ 7370.424873] LustreError: 161499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789046120 with bad export cookie 4020457121176172270 [ 7370.434559] LustreError: 161499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7370.436706] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7370.718687] Lustre: server umount lustre-MDT0001 complete [ 7385.237685] Lustre: server umount lustre-OST0000 complete [ 7411.145196] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [ 7429.540896] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 7475.843436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7476.109281] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7476.133105] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7476.195514] Lustre: lustre-MDT0000: new disk, initializing [ 7476.287452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7476.300960] Lustre: Skipped 1 previous similar message [ 7476.314949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7482.473549] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7495.829110] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7496.009915] Lustre: 170513:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7496.291323] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7501.402421] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7512.603432] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7519.102104] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7528.034701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7528.377844] Lustre: lustre-OST0000: new disk, initializing [ 7528.381282] Lustre: Skipped 1 previous similar message [ 7528.386095] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7528.392346] Lustre: Skipped 2 previous similar messages [ 7528.399483] Lustre: 172482:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7535.561073] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7538.280148] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7538.285581] Lustre: Skipped 1 previous similar message [ 7538.304659] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7538.488924] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7546.636580] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7552.586845] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7552.809101] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7558.415715] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7558.594582] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7564.257176] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7564.271350] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7564.285586] Lustre: Skipped 2 previous similar messages [ 7564.292129] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7564.299802] Lustre: Skipped 3 previous similar messages [ 7566.915942] Lustre: server umount lustre-OST0000 complete [ 7572.960494] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7572.976227] LustreError: Skipped 1 previous similar message [ 7577.455789] Lustre: server umount lustre-MDT0000 complete [ 7579.623219] LustreError: 170525:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7579.646776] LustreError: 170525:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 18 previous similar messages [ 7581.470503] LustreError: 170506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789046331 with bad export cookie 4020457121176189105 [ 7581.476652] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7581.872747] Lustre: server umount lustre-MDT0001 complete [ 7591.897868] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 09:18:58 (1789046338) [ 7599.958967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7606.464971] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7616.614324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7621.877596] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7631.504585] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7639.214735] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7648.239550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7656.856679] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7667.594374] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7687.191779] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 09:20:33 (1789046433) [ 7692.768157] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7692.783742] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7692.809556] Lustre: Skipped 3 previous similar messages [ 7692.824458] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7692.827263] Lustre: Skipped 3 previous similar messages [ 7698.790188] Lustre: server umount lustre-OST0000 complete [ 7709.630555] Lustre: server umount lustre-MDT0000 complete [ 7713.936522] LustreError: 174877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789046463 with bad export cookie 4020457121176189987 [ 7713.947716] LustreError: 174877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7713.949321] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7714.401503] Lustre: server umount lustre-MDT0001 complete [ 7728.079402] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 7731.145456] Key type lgssc unregistered [ 7731.405609] LNet: 178647:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7731.417427] LNetError: 178647:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7731.439606] LNet: Removed LNI 192.168.202.106@tcp [ 7732.452406] Key type .llcrypt unregistered [ 7732.458623] Key type ._llcrypt unregistered [ 7755.537680] Key type ._llcrypt registered [ 7755.547939] Key type .llcrypt registered [ 7755.691735] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 7757.438721] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7757.470777] alg: No test for adler32 (adler32-zlib) [ 7758.711077] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 7759.132024] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 7760.823838] Key type lgssc registered [ 7762.389599] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7771.782621] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 7771.801388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7787.366303] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7792.479362] LustreError: 179886:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.106@tcp: failed processing log, type 4: rc = -110 [ 7797.600797] LustreError: 179886:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7797.621062] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7797.631135] LustreError: MGC192.168.202.106@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7797.651214] LustreError: 179886:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7797.671170] LustreError: 179886:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -22 [ 7797.683191] LustreError: 179886:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 7797.772163] Lustre: server umount lustre-OST0000 complete [ 7797.781735] LustreError: 179886:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7808.485336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7810.118592] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7815.451556] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7820.258788] LustreError: 180386:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7820.270763] LustreError: 180386:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7821.283058] LustreError: 180387:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7825.798830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7826.122251] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7830.557706] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7838.486587] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7845.639475] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7854.838830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7855.171693] LustreError: 182203:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7855.184655] Lustre: 182203:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.202.106@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7855.340855] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7863.781606] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7874.177712] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7877.603434] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7877.612098] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7877.635224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7881.189366] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7881.191113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7881.219829] Lustre: Skipped 1 previous similar message [ 7881.236138] Lustre: Skipped 2 previous similar messages [ 7882.847331] Lustre: server umount lustre-MDT0000 complete [ 7886.307098] LustreError: 182066:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7886.334338] LustreError: 182066:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7887.747171] LustreError: 180368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789046637 with bad export cookie 6116530253593922872 [ 7887.757651] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7888.259836] Lustre: server umount lustre-MDT0001 complete [ 7902.763486] Lustre: server umount lustre-OST0000 complete [ 7913.319767] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [ 7926.365044] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 7974.369640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7974.743207] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7974.785164] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7974.918826] Lustre: lustre-MDT0000: new disk, initializing [ 7975.014356] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7975.030525] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7980.357191] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7992.895643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7993.075489] Lustre: 185733:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7993.178626] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7993.193858] Lustre: Skipped 1 previous similar message [ 7993.327912] Lustre: lustre-MDT0001: new disk, initializing [ 7993.417312] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7993.536971] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7993.548660] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7998.946975] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 8011.914506] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8020.521370] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8028.942406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8029.290517] Lustre: lustre-OST0000: new disk, initializing [ 8029.294603] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8029.304811] Lustre: 187705:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8029.386445] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8035.942701] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8035.955612] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8036.075339] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8037.275898] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 8048.504769] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8054.044239] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8054.283765] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8059.068187] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8059.449559] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8067.045449] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8067.063480] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8067.069968] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8067.086408] Lustre: Skipped 1 previous similar message [ 8072.164648] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8072.174688] Lustre: Skipped 2 previous similar messages [ 8076.255453] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8076.635807] Lustre: server umount lustre-OST0000 complete [ 8082.401238] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8082.427698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8082.434605] Lustre: Skipped 1 previous similar message [ 8087.535195] LustreError: 187347:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8087.565113] LustreError: 187347:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8087.617836] Lustre: server umount lustre-MDT0000 complete [ 8092.594694] LustreError: 185725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789046842 with bad export cookie 6116530253593923593 [ 8092.603704] LustreError: 185725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 8092.614930] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8093.010294] Lustre: server umount lustre-MDT0001 complete [ 8103.566982] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 09:27:30 (1789046850) [ 8124.951940] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 8129.165247] Key type lgssc unregistered [ 8129.663115] LNet: 190651:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8129.668500] LNetError: 190651:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8129.690466] LNet: Removed LNI 192.168.202.106@tcp [ 8130.863888] Key type .llcrypt unregistered [ 8130.867230] Key type ._llcrypt unregistered [ 8151.818432] Key type ._llcrypt registered [ 8151.821200] Key type .llcrypt registered [ 8151.929476] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 8153.332536] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8153.465841] alg: No test for adler32 (adler32-zlib) [ 8154.629930] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 8154.855611] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 8156.583309] Key type lgssc registered [ 8157.994112] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8165.932163] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 8165.970069] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8167.583743] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8172.755206] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 8178.144541] LustreError: 191867:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8178.178275] LustreError: 191867:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8182.937467] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8183.390075] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8188.154922] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 8196.931856] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8203.508876] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8212.526468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8212.720285] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8212.726268] LustreError: 193732:0:(llog.c:1696:llog_backup()) MGC192.168.202.106@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8212.736614] Lustre: 193732:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.202.106@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8212.832871] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8219.454492] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 8228.166782] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8233.454775] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8233.462686] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8233.468379] Lustre: Skipped 1 previous similar message [ 8233.482496] Lustre: Skipped 2 previous similar messages [ 8237.060921] Lustre: server umount lustre-MDT0000 complete [ 8238.569217] LustreError: 191863:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8238.590378] LustreError: 191863:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8240.863346] LustreError: 191849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789046990 with bad export cookie 6709744426588819403 [ 8240.879301] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8240.881589] LustreError: 191849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8241.393530] Lustre: server umount lustre-MDT0001 complete [ 8256.034495] Lustre: server umount lustre-OST0000 complete [ 8268.002085] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [ 8280.700883] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 8325.350840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8325.851141] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8325.881513] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8326.029585] Lustre: lustre-MDT0000: new disk, initializing [ 8326.150107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8326.214851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8332.798426] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 8344.017288] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8344.115932] Lustre: 197262:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8344.155812] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8344.158563] Lustre: Skipped 1 previous similar message [ 8344.161613] LustreError: 198046:0:(llog.c:1696:llog_backup()) MGC192.168.202.106@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8344.167207] LustreError: 198046:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8344.172712] Lustre: 198046:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.106@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8344.187800] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8344.192252] Lustre: Skipped 1 previous similar message [ 8344.281737] Lustre: lustre-MDT0001: new disk, initializing [ 8344.404105] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8344.452785] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8344.458565] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8350.001632] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 8360.480529] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8367.428331] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8376.046871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8376.274437] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8376.277770] Lustre: Skipped 1 previous similar message [ 8376.280200] LustreError: 199207:0:(llog.c:1696:llog_backup()) MGC192.168.202.106@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8376.287782] LustreError: 199207:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8376.291464] Lustre: 199207:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.106@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8376.297196] Lustre: 199207:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8376.311942] Lustre: lustre-OST0000: new disk, initializing [ 8376.315048] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8376.320352] Lustre: 199230:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8376.401536] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8382.562576] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8382.579964] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8382.758789] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8384.689949] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 8395.038554] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8400.519760] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8400.824714] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8406.739880] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8407.036759] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8413.665136] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8413.665930] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8413.681369] LustreError: Skipped 1 previous similar message [ 8413.707865] Lustre: Skipped 2 previous similar messages [ 8413.717120] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8418.790706] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8418.802206] Lustre: Skipped 2 previous similar messages [ 8423.393327] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8423.574246] Lustre: server umount lustre-OST0000 complete [ 8429.023965] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8429.026189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8429.051256] Lustre: Skipped 1 previous similar message [ 8429.065832] Lustre: Skipped 1 previous similar message [ 8433.830134] Lustre: server umount lustre-MDT0000 complete [ 8434.152517] LustreError: 197270:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8434.168148] LustreError: 197270:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8438.658172] LustreError: 197254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789047188 with bad export cookie 6709744426588820117 [ 8438.660857] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8438.665841] LustreError: 197254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8438.962617] Lustre: server umount lustre-MDT0001 complete [ 8450.036917] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 09:33:16 (1789047196) [ 8468.759759] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 8479.962181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8480.387549] LustreError: 202362:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8480.460120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8485.089789] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 8485.860042] LustreError: 202363:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8495.941105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8496.097040] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8502.459323] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 8506.385359] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8515.773882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8516.107269] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8516.119023] Lustre: Skipped 1 previous similar message [ 8522.786859] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 8531.905980] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8532.203442] Lustre: lustre-OST0001: new disk, initializing [ 8532.211113] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8532.219350] Lustre: 204670:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8539.374276] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 8541.807537] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8541.815143] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8541.925051] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8551.147733] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8563.697539] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8566.359574] Lustre: DEBUG MARKER: ADD OST3 [ 8572.181092] loop: module loaded [ 8572.296893] loop0: detected capacity change from 0 to 204800000 [ 8572.723694] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8572.753348] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8572.770545] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8579.952361] Lustre: DEBUG MARKER: STOP OST3 [ 8579.974508] loop0: detected capacity change from 0 to 204800000 [ 8587.306967] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8587.631416] Lustre: lustre-OST0003: new disk, initializing [ 8587.643804] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8587.657578] Lustre: 206716:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8587.847702] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8587.857443] Lustre: Skipped 1 previous similar message [ 8588.917412] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8589.098098] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8589.108734] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8589.152654] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8590.944600] LustreError: 206795:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8591.830292] Lustre: Failing over lustre-OST0003 [ 8591.962744] Lustre: server umount lustre-OST0003 complete [ 8593.863272] LustreError: 203850:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.202.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8593.876493] LustreError: 203850:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8594.442372] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8594.446222] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8594.457968] Lustre: Skipped 1 previous similar message [ 8597.068430] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 8608.998330] Lustre: DEBUG MARKER: START OST3 again [ 8614.314759] LustreError: 203851:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.202.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8614.336629] LustreError: 203851:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 8617.277834] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8617.465769] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8618.749856] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8619.210556] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8619.221722] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8624.730371] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 8648.235722] Lustre: Failing over lustre-OST0003 [ 8648.339539] Lustre: server umount lustre-OST0003 complete [ 8650.220814] LustreError: lustre-OST0003-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8650.223989] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8650.241435] LustreError: Skipped 1 previous similar message [ 8650.246305] LustreError: 203849:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8650.246318] LustreError: 203849:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8650.311864] Lustre: Skipped 1 previous similar message [ 8668.646177] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8668.650263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8668.662226] Lustre: Skipped 2 previous similar messages [ 8668.678813] Lustre: Skipped 3 previous similar messages [ 8673.830712] Lustre: server umount lustre-MDT0000 complete [ 8677.938609] LustreError: 202343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789047427 with bad export cookie 6709744426588821006 [ 8677.945351] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8677.949566] LustreError: 202343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8684.340841] Lustre: server umount lustre-MDT0001 complete [ 8689.600069] Lustre: server umount lustre-OST0000 complete [ 8694.021517] Lustre: server umount lustre-OST0001 complete [ 8702.996914] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [ 8713.928325] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 8757.444937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8757.701267] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8757.779631] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8757.890124] Lustre: lustre-MDT0000: new disk, initializing [ 8758.036785] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8758.050124] Lustre: Skipped 1 previous similar message [ 8758.077868] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8763.714845] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 8776.047699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8776.197450] Lustre: 212281:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8776.310820] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8776.314224] Lustre: Skipped 1 previous similar message [ 8776.386747] Lustre: lustre-MDT0001: new disk, initializing [ 8776.621954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8776.638840] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8783.599178] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 8795.215865] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8802.109634] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8809.978135] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8810.282595] Lustre: lustre-OST0000: new disk, initializing [ 8810.285905] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8810.291582] Lustre: 214254:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8812.181773] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8812.196516] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8812.331731] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8817.597769] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 8827.729431] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8833.986812] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8834.322688] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8839.601304] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8839.842638] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8843.232319] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8843.237324] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8843.264664] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8843.286964] Lustre: Skipped 4 previous similar messages [ 8843.297710] Lustre: Skipped 4 previous similar messages [ 8848.348952] Lustre: server umount lustre-OST0000 complete [ 8862.155133] Lustre: server umount lustre-MDT0000 complete [ 8863.732620] LustreError: 212287:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8863.784021] LustreError: 212287:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 18 previous similar messages [ 8867.495721] LustreError: 212274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789047617 with bad export cookie 6709744426588823295 [ 8867.501325] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8867.505566] LustreError: 212274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8868.325880] Lustre: server umount lustre-MDT0001 complete [ 8872.893230] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 09:40:19 (1789047619) [ 8893.573279] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [ 8910.669993] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 8959.096181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8959.488866] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8959.570893] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8959.820237] Lustre: lustre-MDT0000: new disk, initializing [ 8960.044238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8966.658668] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 8979.493811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8979.759713] Lustre: 218890:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8980.411792] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8987.306535] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 8999.753829] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9007.479569] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9016.849393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9017.347160] Lustre: 220861:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9017.459116] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9017.463915] Lustre: Skipped 4 previous similar messages [ 9018.901364] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9019.079767] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9024.014610] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 9034.287494] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9040.017276] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9040.298444] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9045.482769] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9045.753910] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9050.085970] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9050.090107] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9050.094693] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9050.094701] Lustre: Skipped 5 previous similar messages [ 9050.096668] Lustre: Skipped 3 previous similar messages [ 9050.141746] LustreError: Skipped 1 previous similar message [ 9062.367223] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9062.593257] Lustre: server umount lustre-OST0000 complete [ 9067.489269] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9081.311985] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9081.696077] Lustre: server umount lustre-MDT0000 complete [ 9085.923690] LustreError: 218900:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9085.940979] LustreError: 218900:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9086.260656] LustreError: 218880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789047835 with bad export cookie 6709744426588824184 [ 9086.282438] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9086.283503] LustreError: 218880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9086.762050] Lustre: server umount lustre-MDT0001 complete [ 9096.094849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9100.945934] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 9110.486572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9116.596979] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 9125.735567] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9133.101660] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9140.880425] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9148.609104] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 9158.518492] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9170.916817] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9170.922635] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9170.925488] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9170.935105] Lustre: Skipped 3 previous similar messages [ 9170.948021] Lustre: Skipped 13 previous similar messages [ 9175.295132] Lustre: server umount lustre-OST0000 complete [ 9185.850252] Lustre: server umount lustre-MDT0000 complete [ 9189.849676] LustreError: 223060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789047939 with bad export cookie 6709744426588825066 [ 9189.859820] LustreError: 223060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9189.860282] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9190.278266] Lustre: server umount lustre-MDT0001 complete [ 9300.922269] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 9303.781454] Key type lgssc unregistered [ 9304.201752] LNet: 226741:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9304.213561] LNetError: 226741:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9304.232199] LNet: Removed LNI 192.168.202.106@tcp [ 9305.290175] Key type .llcrypt unregistered [ 9305.294223] Key type ._llcrypt unregistered [ 9334.282477] Key type ._llcrypt registered [ 9334.285680] Key type .llcrypt registered [ 9334.414338] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [ 9352.590368] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 9354.099227] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9354.264706] alg: No test for adler32 (adler32-zlib) [ 9355.378606] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 9355.701865] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 9357.471390] Key type lgssc registered [ 9359.111654] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9400.946625] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 9400.989957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9402.233639] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9402.282193] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9402.421854] Lustre: lustre-MDT0000: new disk, initializing [ 9402.526553] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9402.543571] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9407.998345] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 9421.008027] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9421.246769] Lustre: 230094:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9421.370476] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9421.383810] Lustre: Skipped 1 previous similar message [ 9421.551669] Lustre: lustre-MDT0001: new disk, initializing [ 9421.755966] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9421.835995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9421.854341] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9427.623637] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 9438.818386] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9446.125222] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9454.733661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9455.186419] Lustre: lustre-OST0000: new disk, initializing [ 9455.194888] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9455.205525] Lustre: 232063:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9455.332875] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9461.792498] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 9463.895944] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9463.904392] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9463.990079] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9472.457975] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9478.106235] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9478.339898] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9482.852272] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9483.176783] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9489.898314] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9489.900596] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9489.909787] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9489.942093] Lustre: Skipped 1 previous similar message [ 9491.433295] Lustre: server umount lustre-OST0000 complete [ 9498.593607] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9498.608723] LustreError: Skipped 1 previous similar message [ 9498.614814] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9498.631974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9498.642300] Lustre: Skipped 1 previous similar message [ 9500.130111] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9500.130682] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9501.960464] Lustre: server umount lustre-MDT0000 complete [ 9506.094926] LustreError: 230089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789048255 with bad export cookie 10275408574754035608 [ 9506.101774] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9506.106667] LustreError: 230089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9506.492747] Lustre: server umount lustre-MDT0001 complete [ 9528.823850] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 9541.073876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9541.908434] LustreError: 234998:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9541.935027] LustreError: 234998:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9542.003658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9547.236933] LustreError: 234999:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9547.696105] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 9552.353276] LustreError: 234998:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9557.158878] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9557.474047] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9557.480824] Lustre: Skipped 1 previous similar message [ 9557.672793] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9562.755193] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 9566.443463] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9575.659986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9576.257625] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9585.873362] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 9597.112584] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9597.363810] Lustre: lustre-OST0001: new disk, initializing [ 9597.367874] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9597.380843] Lustre: 237308:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9597.526660] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9604.583056] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 9605.195825] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9605.207512] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9605.288941] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9615.803212] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9627.243865] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9644.168259] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 09:53:11 (1789048391) [ 9646.572424] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9646.584392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9646.590575] Lustre: Skipped 1 previous similar message [ 9646.612676] Lustre: Skipped 3 previous similar messages [ 9652.912835] Lustre: server umount lustre-MDT0000 complete [ 9656.636892] LustreError: 236506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789048406 with bad export cookie 10275408574754036490 [ 9656.646559] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9656.651941] LustreError: 236506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9656.811677] LustreError: 234994:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9656.822786] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9656.842655] LustreError: 234994:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9656.868984] Lustre: Skipped 2 previous similar messages [ 9656.878093] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9656.888918] Lustre: Skipped 5 previous similar messages [ 9657.000699] Lustre: server umount lustre-MDT0001 complete [ 9663.151212] Lustre: server umount lustre-OST0000 complete [ 9667.401879] Lustre: server umount lustre-OST0001 complete [ 9676.229748] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [ 9691.418741] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 9736.659889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9736.954525] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9737.012934] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9737.093249] Lustre: lustre-MDT0000: new disk, initializing [ 9737.165986] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9737.182849] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9741.775292] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 9752.994243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9753.111351] Lustre: 241274:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9753.163776] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9753.169374] Lustre: Skipped 1 previous similar message [ 9753.262324] Lustre: lustre-MDT0001: new disk, initializing [ 9753.483607] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9753.518740] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9758.639132] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 9768.567714] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9775.138192] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9783.029431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9783.325371] Lustre: lustre-OST0000: new disk, initializing [ 9783.328746] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9783.335323] Lustre: 243244:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9784.953764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9784.967403] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9785.147678] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9790.228933] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 9800.906908] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9806.408245] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9806.547943] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9811.317981] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9811.528763] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9816.047324] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9816.053331] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9816.054518] Lustre: Skipped 2 previous similar messages [ 9816.055165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9819.788590] Lustre: server umount lustre-OST0000 complete [ 9825.248911] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9825.272387] LustreError: Skipped 1 previous similar message [ 9831.316372] Lustre: server umount lustre-MDT0000 complete [ 9831.393932] LustreError: 241281:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9831.412325] LustreError: 241281:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9835.334045] LustreError: 243243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789048584 with bad export cookie 10275408574754038625 [ 9835.346903] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9835.349306] LustreError: 243243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9835.730258] Lustre: server umount lustre-MDT0001 complete [ 9844.113935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9844.645431] LustreError: 245460:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9844.723171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9844.726291] Lustre: Skipped 2 previous similar messages [ 9849.625380] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 9859.512137] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9864.901842] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 9872.967330] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9878.842149] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9885.853982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9892.340437] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 9900.663938] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10048.999491] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10049.016167] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10049.025304] Lustre: Skipped 2 previous similar messages [10049.035182] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10049.041851] Lustre: Skipped 4 previous similar messages [10062.303207] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10062.677145] Lustre: server umount lustre-OST0000 complete [10073.847939] Lustre: server umount lustre-MDT0000 complete [10078.688890] LustreError: 247977:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10078.713870] LustreError: 247977:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10079.011284] LustreError: 245441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789048828 with bad export cookie 10275408574754039507 [10079.015033] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10079.026459] LustreError: 245441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10079.362493] Lustre: server umount lustre-MDT0001 complete [10167.939388] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [10171.504223] Key type lgssc unregistered [10171.893018] LNet: 249079:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10171.899981] LNetError: 249079:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10171.928500] LNet: Removed LNI 192.168.202.106@tcp [10173.201287] Key type .llcrypt unregistered [10173.208680] Key type ._llcrypt unregistered [10201.242564] Key type ._llcrypt registered [10201.244820] Key type .llcrypt registered [10201.347090] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [10219.135684] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [10220.225888] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10220.446099] alg: No test for adler32 (adler32-zlib) [10221.644515] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [10221.828636] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [10223.487494] Key type lgssc registered [10224.841923] Lustre: Echo OBD driver; http://www.lustre.org/ [10282.348167] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [10293.816109] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [10293.855469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10295.109386] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10295.146721] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10295.258168] Lustre: lustre-MDT0000: new disk, initializing [10295.436253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10295.501352] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10301.183673] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [10312.467131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10312.626388] Lustre: 253147:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10312.739818] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10312.750352] Lustre: Skipped 1 previous similar message [10312.849574] Lustre: lustre-MDT0001: new disk, initializing [10312.992674] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10313.051275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10313.063884] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10317.655841] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [10323.186419] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10330.510948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10331.073135] Lustre: lustre-OST0000: new disk, initializing [10331.086061] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10331.099965] Lustre: 254779:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10331.228209] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10337.355597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10337.373587] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10337.560979] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10338.264651] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [10348.768794] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10349.055881] Lustre: lustre-OST0001: new disk, initializing [10349.066902] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10349.090249] Lustre: 255653:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10349.252142] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10356.377639] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10356.401521] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10356.595277] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10357.009742] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [10368.852421] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10378.060440] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10392.547212] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10392.552718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10392.582469] Lustre: Skipped 1 previous similar message [10392.599182] Lustre: Skipped 3 previous similar messages [10397.667858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10397.678700] Lustre: Skipped 3 previous similar messages [10402.785452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10402.796632] Lustre: Skipped 2 previous similar messages [10405.344941] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10407.579179] Lustre: server umount lustre-MDT0000 complete [10411.682708] LustreError: 253142:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789049161 with bad export cookie 3833064008957930735 [10411.684986] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10411.704035] LustreError: 253142:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10412.349904] Lustre: server umount lustre-MDT0001 complete [10427.108347] Lustre: server umount lustre-OST0000 complete [10428.959993] Lustre: 250654:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789049162/real 1789049162] req@ffff994f48f17480 x1875953813898240/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789049178 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10428.994631] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10429.006102] Lustre: Skipped 2 previous similar messages [10431.455517] Lustre: 250652:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789049165/real 1789049165] req@ffff994f48f17100 x1875953813898496/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789049181 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10431.757743] Lustre: server umount lustre-OST0001 complete [10448.543891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10460.424472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10470.027414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10480.914717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10503.564504] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [10513.787168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10513.799554] Lustre: lustre-MDT0000: reset Object Index mappings [10514.139675] LustreError: 260668:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10514.151285] LustreError: 260668:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [10514.213842] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10519.016202] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [10519.531390] LustreError: 260669:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10523.631122] LustreError: 260668:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10528.021031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10528.039641] Lustre: lustre-MDT0001: reset Object Index mappings [10528.708635] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10534.468941] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [10537.734565] Lustre: 261811:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10545.276902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10545.604609] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10552.460758] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [10553.782143] LustreError: 262165:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10558.903865] LustreError: 262165:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10558.930313] LustreError: 262165:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10558.933169] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10562.997413] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10568.743377] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10571.634250] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [10581.730640] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10586.976793] Lustre: 263681:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10594.278147] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10594.291273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10594.293731] Lustre: Skipped 2 previous similar messages [10594.311684] Lustre: Skipped 2 previous similar messages [10594.596641] Lustre: server umount lustre-MDT0000 complete [10599.395480] LustreError: 260668:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10599.444450] LustreError: 260668:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10599.650728] LustreError: 260648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789049349 with bad export cookie 3833064008957937525 [10599.658296] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10599.666476] LustreError: 260648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10600.369129] Lustre: server umount lustre-MDT0001 complete [10616.665837] Lustre: server umount lustre-OST0000 complete [10620.896133] Lustre: 250653:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789049354/real 1789049354] req@ffff994f48f17480 x1875953813950080/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789049370 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10620.943547] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10620.973905] Lustre: Skipped 1 previous similar message [10621.725649] Lustre: server umount lustre-OST0001 complete [10650.029366] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 10:09:56 (1789049396) [10671.470591] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [10686.868425] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [10729.354813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10729.562536] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10729.587217] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10729.681285] Lustre: lustre-MDT0000: new disk, initializing [10729.757829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10729.762696] Lustre: Skipped 1 previous similar message [10729.776360] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10734.436866] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [10748.193112] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10748.321529] Lustre: 267978:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10748.426788] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10748.436930] Lustre: Skipped 1 previous similar message [10748.575483] Lustre: lustre-MDT0001: new disk, initializing [10748.746431] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10748.766550] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10754.447298] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [10765.554927] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10771.852352] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10779.731684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10780.139380] Lustre: lustre-OST0000: new disk, initializing [10780.141997] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10780.153603] Lustre: 269948:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10782.224710] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10782.241185] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10782.445949] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10787.564985] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [10799.688201] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10805.940625] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10806.266427] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10811.935962] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10812.282777] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10818.539198] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10818.542860] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10818.543643] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10818.553013] Lustre: Skipped 1 previous similar message [10818.560821] Lustre: Skipped 1 previous similar message [10828.794158] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10828.804382] Lustre: Skipped 3 previous similar messages [10829.282074] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10829.570673] Lustre: server umount lustre-OST0000 complete [10835.942326] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10835.950458] LustreError: Skipped 1 previous similar message [10835.955609] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10840.726525] Lustre: server umount lustre-MDT0000 complete [10844.137558] LustreError: 267985:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10844.170304] LustreError: 267985:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10845.998861] LustreError: 267972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789049595 with bad export cookie 3833064008957939611 [10846.022414] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10846.026031] LustreError: 267972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10846.472724] Lustre: server umount lustre-MDT0001 complete [10869.169612] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [10882.353974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10882.753369] LustreError: 272885:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10882.796307] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10882.800762] Lustre: Skipped 2 previous similar messages [10888.817123] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [10899.752264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10905.922650] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [10910.509550] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10919.553731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10927.136686] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [10938.650114] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10938.938189] Lustre: lustre-OST0001: new disk, initializing [10938.948147] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10938.956124] Lustre: 275198:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10940.155155] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10940.168683] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10940.313404] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10948.481385] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [10961.022643] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10967.264098] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10974.165785] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10974.286968] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10991.633289] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10991.648819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10991.679668] Lustre: Skipped 3 previous similar messages [10991.696354] Lustre: Skipped 5 previous similar messages [10995.564923] Lustre: server umount lustre-MDT0000 complete [10996.715239] LustreError: 275726:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10996.744139] LustreError: 275726:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [11000.495185] LustreError: 272866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789049750 with bad export cookie 3833064008957940500 [11000.502483] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11000.510767] LustreError: 272866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11000.817730] Lustre: server umount lustre-MDT0001 complete [11016.063538] Lustre: server umount lustre-OST0000 complete [11018.207454] Lustre: 250652:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789049751/real 1789049751] req@ffff994f7e902d80 x1875953814073856/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789049767 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11021.494067] Lustre: server umount lustre-OST0001 complete [11045.224688] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [11056.760568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11057.133894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11057.147341] Lustre: Skipped 3 previous similar messages [11062.326178] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [11072.043917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11077.416645] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [11080.416890] Lustre: 279685:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11089.796322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11099.072479] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [11101.700415] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [11110.344051] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11116.049541] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [11119.982274] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [11129.130813] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11133.372318] Lustre: 281561:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11139.371137] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [11139.581969] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [11154.904230] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 10:18:21 (1789049901) [11171.300238] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11171.305844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11171.325110] Lustre: Skipped 4 previous similar messages [11171.343565] Lustre: Skipped 4 previous similar messages [11176.399386] Lustre: server umount lustre-MDT0000 complete [11176.436493] LustreError: 279545:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11176.474564] LustreError: 279545:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [11179.903904] LustreError: 278525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789049929 with bad export cookie 3833064008957947178 [11179.922941] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11194.335470] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11194.727927] Lustre: server umount lustre-MDT0001 complete [11213.279225] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11213.436404] Lustre: server umount lustre-OST0000 complete [11218.077865] Lustre: server umount lustre-OST0001 complete [11226.124561] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [11242.018836] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [11302.422413] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [11314.540775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11314.925659] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11314.990349] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11315.155285] Lustre: lustre-MDT0000: new disk, initializing [11315.328742] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11315.337237] Lustre: Skipped 3 previous similar messages [11315.352490] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11320.863826] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [11331.859551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11332.066574] Lustre: 286134:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11332.562486] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11337.493792] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [11350.776827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11351.117338] Lustre: 287765:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11352.427925] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11359.178100] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [11371.555167] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11371.754186] Lustre: 288639:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11373.418243] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11373.432185] Lustre: Skipped 1 previous similar message [11373.575784] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11379.117129] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [11391.178855] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11395.249896] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11395.257598] Lustre: Skipped 1 previous similar message [11412.668430] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 10:22:40 (1789050160) [11419.624997] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11419.631081] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11419.659425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11419.667546] Lustre: Skipped 6 previous similar messages [11419.690777] Lustre: Skipped 9 previous similar messages [11420.953609] Lustre: server umount lustre-MDT0000 complete [11425.003510] LustreError: 286127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789050174 with bad export cookie 3833064008957954871 [11425.011186] LustreError: 286127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11425.019481] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11425.329815] Lustre: server umount lustre-MDT0001 complete [11440.401077] Lustre: server umount lustre-OST0000 complete [11454.389587] Lustre: server umount lustre-OST0001 complete [11464.060480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11464.593340] LustreError: 290885:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11464.619038] LustreError: 290885:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [11470.152851] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [11481.351260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11487.571562] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [11495.901488] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11503.036784] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11511.765823] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11519.258735] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [11528.087715] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11558.974167] loop1: detected capacity change from 0 to 204800000 [11559.355349] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11559.378868] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11559.412127] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11565.499189] loop1: detected capacity change from 0 to 204800000 [11572.795642] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11572.906696] LustreError: 290876:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs4463: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11572.914510] LustreError: 290876:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs4463 [11572.921068] LustreError: 290876:0:(mgs_handler.c:517:mgs_target_reg()) fs4463-MDT0001: Error checking target index: rc = -13 [11572.932657] LustreError: fs4463-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11572.938543] LustreError: 294627:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [11572.945256] LustreError: 294627:0:(tgt_mount.c:2128:server_put_super()) no obd fs4463-MDT0001 [11572.952251] LustreError: 294627:0:(tgt_mount.c:116:server_deregister_mount()) fs4463-MDT0001 not registered [11572.998280] Lustre: server umount fs4463-MDT0001 complete [11573.000549] LustreError: 294627:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11582.474817] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11582.547338] Lustre: Setting parameter fs4463-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs4463-MDT0001 [11582.583985] Lustre: srv-fs4463-MDT0001: No data found on store. Initialize space. [11582.586401] Lustre: Skipped 4 previous similar messages [11582.650732] Lustre: fs4463-MDT0001: new disk, initializing [11582.653585] Lustre: Skipped 3 previous similar messages [11582.760290] Lustre: srv-fs4463-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11582.770288] Lustre: 295094:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs4463-MDT0001-mdtlov: can't create new llog: rc = -115 [11582.779689] LustreError: 295094:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs4463-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11587.422752] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [11594.900782] Lustre: server umount fs4463-MDT0001 complete [11606.500544] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11610.913884] loop1: detected capacity change from 0 to 204800000 [11611.652963] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11618.309967] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11618.657462] Lustre: 296662:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs4463-MDT0001-mdtlov: can't create new llog: rc = -115 [11618.669115] LustreError: 296662:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs4463-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11623.603574] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [11630.407631] Lustre: server umount fs4463-MDT0001 complete [11641.803821] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.6@tcp (stopping) [11642.343819] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11642.350823] Lustre: Skipped 1 previous similar message [11647.531894] Lustre: server umount lustre-MDT0000 complete [11655.933538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11656.079609] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11661.502827] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [11661.823962] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11670.511965] loop1: detected capacity change from 0 to 204800000 [11676.812553] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11677.123806] Lustre: 299112:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs4463-MDT0001-mdtlov: can't create new llog: rc = -115 [11677.130988] LustreError: 299112:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs4463-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11681.741605] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [11687.551592] Lustre: server umount fs4463-MDT0001 complete [11716.065420] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11728.351900] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11728.676528] Lustre: server umount lustre-OST0000 complete [11733.473596] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11733.481483] LustreError: Skipped 1 previous similar message [11738.846316] Lustre: server umount lustre-MDT0000 complete [11742.952657] LustreError: 295638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789050492 with bad export cookie 3833064008957957552 [11742.961482] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11742.963539] LustreError: 295638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11743.366875] Lustre: server umount lustre-MDT0001 complete [11755.084786] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [11757.758661] Key type lgssc unregistered [11758.122809] LNet: 301458:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11758.132324] LNetError: 301458:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11758.147385] LNet: Removed LNI 192.168.202.106@tcp [11758.835161] Key type .llcrypt unregistered [11758.837777] Key type ._llcrypt unregistered [11780.882440] Key type ._llcrypt registered [11780.884667] Key type .llcrypt registered [11780.982242] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [11782.254167] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11782.271075] alg: No test for adler32 (adler32-zlib) [11783.434927] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [11783.680203] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [11785.393412] Key type lgssc registered [11786.468878] Lustre: Echo OBD driver; http://www.lustre.org/ [11794.185258] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [11794.202399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11795.774069] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11800.417866] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [11806.180603] LustreError: 302725:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11806.201697] LustreError: 302725:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11809.047419] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11809.443791] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11814.200874] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [11821.900277] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11827.412498] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11834.628519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11835.043553] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11840.761663] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [11841.219656] LustreError: 304553:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11846.630251] LustreError: 304555:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11847.799897] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11850.576609] LustreError: 304939:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11855.772468] LustreError: 304553:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11855.794835] LustreError: 304553:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11864.042407] LustreError: 304555:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11864.065848] LustreError: 304555:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [11868.918164] loop1: detected capacity change from 0 to 204800000 [11868.995710] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11869.012508] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11869.026230] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11873.469766] loop1: detected capacity change from 0 to 204800000 [11878.201759] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11878.377434] Lustre: fs28581-OST0000: new disk, initializing [11878.381797] Lustre: srv-fs28581-OST0000: No data found on store. Initialize space. [11878.387741] Lustre: 306462:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11878.466290] Lustre: fs28581-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11881.369377] LustreError: 304554:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11881.388864] LustreError: 304554:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [11884.299559] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [11891.933942] Lustre: server umount fs28581-OST0000 complete [11906.140575] loop1: detected capacity change from 0 to 204800000 [11911.544929] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11911.691898] Lustre: fs28581-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11915.162199] LustreError: 305227:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11915.188330] LustreError: 305227:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 20 previous similar messages [11917.510445] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [11918.325650] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11926.065212] loop2: detected capacity change from 0 to 204800000 [11926.229376] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11926.236851] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11926.245544] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11933.195673] loop2: detected capacity change from 0 to 204800000 [11933.688626] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11939.582771] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11939.657189] LustreError: fs28581-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11939.671630] LustreError: fs28581-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11939.682358] LustreError: 309501:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [11939.695062] LustreError: 309501:0:(tgt_mount.c:2128:server_put_super()) no obd fs28581-OST0001 [11939.703739] LustreError: 309501:0:(tgt_mount.c:116:server_deregister_mount()) fs28581-OST0001 not registered [11939.734197] Lustre: server umount fs28581-OST0001 complete [11939.741807] LustreError: 309501:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11950.609794] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11950.865233] Lustre: fs28581-OST0001: new disk, initializing [11950.869977] Lustre: srv-fs28581-OST0001: No data found on store. Initialize space. [11950.881232] Lustre: 310055:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11950.956507] Lustre: fs28581-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11956.803936] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [11964.039649] Lustre: server umount fs28581-OST0000 complete [11972.118424] Lustre: server umount fs28581-OST0001 complete [11979.773438] LustreError: 304553:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11979.806477] LustreError: 304553:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 37 previous similar messages [11982.113396] loop1: detected capacity change from 0 to 204800000 [11987.125500] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11987.280455] Lustre: fs28581-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11993.116301] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [11999.325961] loop2: detected capacity change from 0 to 204800000 [12005.384837] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12012.464560] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [12021.636479] Lustre: server umount fs28581-OST0000 complete [12030.304619] Lustre: server umount fs28581-OST0001 complete [12043.232490] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12043.237957] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12043.246406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12046.308911] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12046.312035] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12046.324370] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12046.347981] Lustre: Skipped 1 previous similar message [12049.412945] Lustre: server umount lustre-OST0000 complete [12055.008851] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12055.024673] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12055.041481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12060.090175] Lustre: server umount lustre-MDT0000 complete [12063.682671] LustreError: 302708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789050813 with bad export cookie 8803207547384799276 [12063.686117] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12063.696055] LustreError: 302708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12064.365334] Lustre: server umount lustre-MDT0001 complete [12076.429704] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [12078.971345] Key type lgssc unregistered [12079.335433] LNet: 315979:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12079.340064] LNetError: 315979:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12079.359609] LNet: Removed LNI 192.168.202.106@tcp [12080.269162] Key type .llcrypt unregistered [12080.270620] Key type ._llcrypt unregistered [12112.493662] Key type ._llcrypt registered [12112.496270] Key type .llcrypt registered [12112.592609] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [12128.897452] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [12129.920802] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12129.966618] alg: No test for adler32 (adler32-zlib) [12131.144304] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [12131.530855] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [12133.225476] Key type lgssc registered [12134.381486] Lustre: Echo OBD driver; http://www.lustre.org/ [12167.618029] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [12167.674678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12168.940507] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12168.961639] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12169.014624] Lustre: lustre-MDT0000: new disk, initializing [12169.063482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12169.075576] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12173.653570] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [12185.503589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12185.655563] Lustre: 319839:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12185.722977] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12185.729426] Lustre: Skipped 1 previous similar message [12185.868454] Lustre: lustre-MDT0001: new disk, initializing [12186.038380] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12186.092418] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12186.113873] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12191.854123] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [12202.307960] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12208.555738] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12215.690763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12215.955169] Lustre: lustre-OST0000: new disk, initializing [12215.958461] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12215.962876] Lustre: 321810:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12216.025293] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12222.298752] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [12223.083142] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12223.090895] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12223.281162] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12232.930600] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12234.864507] Lustre: Mounted lustre-client - version 2.17.58_39_gf941a14 [12235.876392] Lustre: Unmounted lustre-client [12245.477441] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12245.498445] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12250.599199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12250.612522] Lustre: Skipped 2 previous similar messages [12255.727399] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12255.732829] Lustre: Skipped 1 previous similar message [12259.295662] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12259.517163] Lustre: server umount lustre-OST0000 complete [12265.957901] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12265.962037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12265.986904] Lustre: Skipped 2 previous similar messages [12266.010661] Lustre: Skipped 1 previous similar message [12271.072552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12271.083929] Lustre: Skipped 1 previous similar message [12278.244828] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12278.429264] Lustre: server umount lustre-MDT0000 complete [12281.922876] LustreError: 319832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789051031 with bad export cookie 13198729286155957718 [12281.929536] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12281.931661] LustreError: 319832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12282.227750] Lustre: server umount lustre-MDT0001 complete [12288.230421] LNet: 1 local NIs in recovery (showing 1): 192.168.202.106@tcp [12294.011505] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [12296.622836] Key type lgssc unregistered [12296.900808] LNet: 323972:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12296.915572] LNetError: 323972:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12367.592333] LNet: Removed LNI 192.168.202.106@tcp [12368.440698] Key type .llcrypt unregistered [12368.445689] Key type ._llcrypt unregistered [12386.616398] Key type ._llcrypt registered [12386.619042] Key type .llcrypt registered [12386.702910] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [12404.536728] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [12405.739807] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12405.991566] alg: No test for adler32 (adler32-zlib) [12407.049349] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [12407.222717] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [12408.871434] Key type lgssc registered [12410.268094] Lustre: Echo OBD driver; http://www.lustre.org/ [12448.659547] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 10:39:55 (1789051195) [12456.077198] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [12456.108425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12457.353433] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12457.420990] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12457.539974] Lustre: lustre-MDT0000: new disk, initializing [12457.681872] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12457.694897] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12462.993136] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [12474.312227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12474.455477] Lustre: 327190:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12474.495361] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12474.498879] Lustre: Skipped 1 previous similar message [12474.583346] Lustre: lustre-MDT0001: new disk, initializing [12474.711251] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12474.789406] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12474.809445] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12480.409878] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [12490.657801] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12497.559950] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12505.138560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12505.602834] Lustre: lustre-OST0000: new disk, initializing [12505.606861] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12505.625180] Lustre: 329160:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12505.750333] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12512.304717] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12512.318333] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [12512.323814] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12512.454293] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12523.015991] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12543.458660] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12543.459634] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12543.473535] LustreError: Skipped 1 previous similar message [12543.481216] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12543.488949] Lustre: Skipped 1 previous similar message [12546.817803] Lustre: server umount lustre-OST0000 complete [12551.649703] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12551.658403] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12551.671579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12551.677771] Lustre: Skipped 1 previous similar message [12553.702462] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12553.703766] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12553.725193] Lustre: Skipped 1 previous similar message [12557.354590] Lustre: server umount lustre-MDT0000 complete [12562.259709] LustreError: 327182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789051311 with bad export cookie 17642739129328620596 [12562.261486] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12562.276823] LustreError: 327182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12562.865759] Lustre: server umount lustre-MDT0001 complete [12575.573307] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [12578.401661] Key type lgssc unregistered [12578.834903] LNet: 331065:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12578.849930] LNetError: 331065:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12578.878193] LNet: Removed LNI 192.168.202.106@tcp [12579.820764] Key type .llcrypt unregistered [12579.824925] Key type ._llcrypt unregistered [12604.649651] Key type ._llcrypt registered [12604.652660] Key type .llcrypt registered [12604.776552] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [12605.940870] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12606.003682] alg: No test for adler32 (adler32-zlib) [12607.209924] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [12607.479246] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [12609.143223] Key type lgssc registered [12610.492850] Lustre: Echo OBD driver; http://www.lustre.org/ [12618.237890] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [12618.256334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12619.703863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12624.017379] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [12629.989149] LustreError: 332336:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12630.020836] LustreError: 332336:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12633.203839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12633.570484] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12638.905707] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [12647.812990] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12654.288788] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12661.471162] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12661.872711] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12668.096586] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [12676.222158] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12690.001455] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12691.803495] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12699.599835] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.6@tcp (stopping) [12700.128544] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12700.131466] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12700.134345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12700.183813] Lustre: Skipped 2 previous similar messages [12701.897063] Lustre: server umount lustre-MDT0000 complete [12704.666996] LustreError: 333578:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12705.723366] LustreError: 332317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789051455 with bad export cookie 5776050891830993095 [12705.744321] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12705.750129] LustreError: 332317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12709.783939] LustreError: 333578:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12709.807163] LustreError: 333578:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12709.808159] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.6@tcp (stopping) [12709.833318] Lustre: Skipped 2 previous similar messages [12710.377800] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12714.911508] LustreError: 332331:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12714.912076] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.6@tcp (stopping) [12714.929026] LustreError: 332331:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12714.945735] Lustre: Skipped 1 previous similar message [12720.022375] LustreError: 332331:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12720.027055] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.6@tcp (stopping) [12720.043486] LustreError: 332331:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12720.063828] Lustre: Skipped 2 previous similar messages [12720.095119] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12720.416342] Lustre: server umount lustre-MDT0001 complete [12724.116605] Lustre: server umount lustre-OST0000 complete [12734.106945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12739.075588] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [12741.743215] Lustre: 336521:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.106@tcp [12743.746922] Lustre: server umount MGS complete [12751.455249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12751.766771] LustreError: 337093:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12751.843025] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12756.264531] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [12766.070027] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12766.440947] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12771.423472] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [12781.073340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12781.493149] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12788.561656] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [12796.692467] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12799.074845] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12806.331563] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12808.079391] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12816.352900] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12816.357767] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12816.372865] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12821.346949] Lustre: server umount lustre-OST0000 complete [12827.619186] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12827.635169] Lustre: Skipped 1 previous similar message [12831.851640] Lustre: server umount lustre-MDT0000 complete [12832.743639] LustreError: 337093:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12832.756665] LustreError: 337093:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12835.824603] LustreError: 337074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789051585 with bad export cookie 5776050891830993928 [12835.830688] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12835.841695] LustreError: 337074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12836.145849] Lustre: server umount lustre-MDT0001 complete [12851.190539] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [12854.234164] Key type lgssc unregistered [12854.527588] LNet: 340529:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12854.533926] LNetError: 340529:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12855.590836] LNet: Removed LNI 192.168.202.106@tcp [12856.315516] Key type .llcrypt unregistered [12856.320886] Key type ._llcrypt unregistered [12875.540466] Key type ._llcrypt registered [12875.542159] Key type .llcrypt registered [12875.659827] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [12894.176853] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [12895.615745] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12895.821709] alg: No test for adler32 (adler32-zlib) [12896.843646] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [12897.187432] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [12898.912110] Key type lgssc registered [12900.621579] Lustre: Echo OBD driver; http://www.lustre.org/ [12946.308621] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [12946.375984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12947.642697] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12947.677364] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12947.777796] Lustre: lustre-MDT0000: new disk, initializing [12947.922341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12947.960062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12952.928917] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [12963.525254] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12963.619637] Lustre: 343688:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12963.659476] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12963.662196] Lustre: Skipped 1 previous similar message [12963.721241] Lustre: lustre-MDT0001: new disk, initializing [12963.816092] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12963.863132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12963.873971] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12968.574599] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [12978.234131] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12983.902237] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12990.676142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12990.913918] Lustre: lustre-OST0000: new disk, initializing [12990.917204] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12990.923741] Lustre: 345654:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12990.996169] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12997.462904] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [13001.281046] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13001.303725] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13001.415579] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13007.295544] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13012.546759] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13012.879206] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13018.213533] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13018.445804] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13022.177473] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13022.195968] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13022.230526] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13026.953586] Lustre: server umount lustre-OST0000 complete [13032.418893] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13032.423031] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13032.446781] Lustre: Skipped 2 previous similar messages [13032.459139] Lustre: Skipped 2 previous similar messages [13037.536830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13037.553194] Lustre: Skipped 1 previous similar message [13041.639783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13041.650819] Lustre: Skipped 1 previous similar message [13045.219967] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13045.628141] Lustre: server umount lustre-MDT0000 complete [13050.935127] LustreError: 343680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789051800 with bad export cookie 10381945535894877497 [13050.946117] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13050.956815] LustreError: 343680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13051.505847] Lustre: server umount lustre-MDT0001 complete [13061.668153] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 10:50:08 (1789051808) [13079.298587] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [13091.211978] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [13132.214810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13132.574659] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13132.657976] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13132.758665] Lustre: lustre-MDT0000: new disk, initializing [13132.871982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13132.894138] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13138.588869] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [13150.324246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13150.501244] Lustre: 350420:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13150.584753] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13150.591674] Lustre: Skipped 1 previous similar message [13150.692826] Lustre: lustre-MDT0001: new disk, initializing [13150.817576] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13150.896325] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13150.899414] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13155.855818] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [13166.870475] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13173.470613] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13181.991821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13182.283483] Lustre: lustre-OST0000: new disk, initializing [13182.291877] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13182.297732] Lustre: 352390:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13182.375052] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13189.502551] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [13191.270917] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13191.279230] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13191.380100] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13199.329658] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13237.736886] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13237.742791] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13237.748144] LustreError: Skipped 2 previous similar messages [13237.753329] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13237.755957] Lustre: Skipped 1 previous similar message [13237.772500] Lustre: Skipped 1 previous similar message [13240.764140] Lustre: server umount lustre-OST0000 complete [13247.969502] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13247.972182] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13247.984193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13247.987327] Lustre: Skipped 1 previous similar message [13247.996937] Lustre: Skipped 1 previous similar message [13250.968268] Lustre: server umount lustre-MDT0000 complete [13253.089729] LustreError: 350425:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13253.101048] LustreError: 350425:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [13254.986731] LustreError: 350412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789052004 with bad export cookie 10381945535894878379 [13254.999556] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13255.002529] LustreError: 350412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13255.364641] Lustre: server umount lustre-MDT0001 complete [13267.942753] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [13270.871599] Key type lgssc unregistered [13271.120873] LNet: 354561:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13271.144536] LNetError: 354561:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13271.173731] LNet: Removed LNI 192.168.202.106@tcp [13272.017611] Key type .llcrypt unregistered [13272.019477] Key type ._llcrypt unregistered [13368.177160] Key type ._llcrypt registered [13368.179717] Key type .llcrypt registered [13368.361629] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [13368.702500] Key type .llcrypt unregistered [13368.707034] Key type ._llcrypt unregistered [13459.473714] Key type ._llcrypt registered [13459.481404] Key type .llcrypt registered [13459.670552] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [13459.966184] Key type .llcrypt unregistered [13459.969695] Key type ._llcrypt unregistered [13494.730820] Key type ._llcrypt registered [13494.732531] Key type .llcrypt registered [13494.817621] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [13495.113704] Key type .llcrypt unregistered [13495.115952] Key type ._llcrypt unregistered [13527.384933] Key type ._llcrypt registered [13527.387650] Key type .llcrypt registered [13527.458431] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [13527.673331] Key type .llcrypt unregistered [13527.674914] Key type ._llcrypt unregistered [13609.710350] Key type ._llcrypt registered [13609.711837] Key type .llcrypt registered [13609.810271] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [13609.991337] Key type .llcrypt unregistered [13609.993127] Key type ._llcrypt unregistered [13644.159408] Key type ._llcrypt registered [13644.166292] Key type .llcrypt registered [13644.289886] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [13644.601560] Key type .llcrypt unregistered [13644.610180] Key type ._llcrypt unregistered [13676.159815] Key type ._llcrypt registered [13676.161861] Key type .llcrypt registered [13676.270519] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [13676.514329] Key type .llcrypt unregistered [13676.515675] Key type ._llcrypt unregistered [13709.533840] Key type ._llcrypt registered [13709.536892] Key type .llcrypt registered [13709.638140] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [13709.905701] Key type .llcrypt unregistered [13709.908443] Key type ._llcrypt unregistered [13742.794759] Key type ._llcrypt registered [13742.798810] Key type .llcrypt registered [13742.926283] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [13743.296381] Key type .llcrypt unregistered [13743.299972] Key type ._llcrypt unregistered [13782.586279] Key type ._llcrypt registered [13782.588174] Key type .llcrypt registered [13782.697264] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [13782.965407] Key type .llcrypt unregistered [13782.967029] Key type ._llcrypt unregistered [13850.463515] Key type ._llcrypt registered [13850.465495] Key type .llcrypt registered [13850.639048] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [13851.188507] Key type .llcrypt unregistered [13851.190105] Key type ._llcrypt unregistered [13885.391995] Key type ._llcrypt registered [13885.396708] Key type .llcrypt registered [13885.571131] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [13885.977160] Key type .llcrypt unregistered [13885.980160] Key type ._llcrypt unregistered [13916.975827] Key type ._llcrypt registered [13916.980426] Key type .llcrypt registered [13917.083761] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [13917.378677] Key type .llcrypt unregistered [13917.381687] Key type ._llcrypt unregistered [13985.654610] Key type ._llcrypt registered [13985.658132] Key type .llcrypt registered [13985.739330] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [14002.678766] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [14003.952976] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14003.994147] alg: No test for adler32 (adler32-zlib) [14005.048403] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [14005.239199] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [14006.903223] Key type lgssc registered [14008.439231] Lustre: Echo OBD driver; http://www.lustre.org/ [14048.845850] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [14048.912355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14050.188589] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [14050.227306] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [14050.342442] Lustre: tst250fs-MDT0000: new disk, initializing [14050.486495] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14050.511290] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14057.761879] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [14070.615804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14070.750520] Lustre: 374877:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14070.791276] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [14070.794801] Lustre: Skipped 1 previous similar message [14070.901148] Lustre: tst250fs-MDT0001: new disk, initializing [14071.070554] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14071.128760] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14071.146533] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14076.785517] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [14081.906476] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [14088.490137] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14088.852295] Lustre: tst250fs-OST0000: new disk, initializing [14088.855919] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [14088.868869] Lustre: 376509:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14088.991126] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14094.966678] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14094.983589] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14095.215558] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14096.183339] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [14107.445550] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14107.664259] Lustre: tst250fs-OST0001: new disk, initializing [14107.667466] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [14107.672514] Lustre: 377380:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14107.755806] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [14113.382317] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [14113.389768] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [14113.491113] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [14114.157661] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [14120.319471] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [14122.724530] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [14122.733677] Lustre: Skipped 1 previous similar message [14144.488722] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14144.499901] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14144.507082] Lustre: Skipped 1 previous similar message [14144.527546] Lustre: Skipped 1 previous similar message [14149.621784] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14149.631860] Lustre: Skipped 3 previous similar messages [14150.658851] Lustre: server umount tst250fs-MDT0000 complete [14154.485528] LustreError: 374869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789052904 with bad export cookie 14733642104326887459 [14154.493356] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14154.720451] Lustre: server umount tst250fs-MDT0001 complete [14169.369610] Lustre: server umount tst250fs-OST0000 complete [14171.616623] Lustre: 373095:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789052904/real 1789052904] req@ffff994f45daf480 x1875957780652032/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789052920 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14171.656284] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14171.685167] Lustre: Skipped 2 previous similar messages [14173.920906] Lustre: server umount tst250fs-OST0001 complete [14196.847482] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [14207.691767] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [14239.364967] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 11:09:46 (1789052986) [14241.317481] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [14262.065496] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [14274.170484] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [14310.325689] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14007 sec ========= 11:10:57 (1789053057) [14312.998289] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 11:10:59 (1789053059) === [14317.524678] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 11:11:04 (1789053064) === [14369.080731] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [14372.455719] Key type lgssc unregistered [14372.923473] LNet: 386531:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14372.928284] LNetError: 386531:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14372.955187] LNet: Removed LNI 192.168.202.106@tcp [14373.992161] Key type .llcrypt unregistered [14374.000209] Key type ._llcrypt unregistered