[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 557604869 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002373] x2apic enabled [ 0.003000] Switched APIC routing to physical x2apic. [ 0.003000] kvm-guest: setup PV IPIs [ 0.005558] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.006000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.006022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.007011] pid_max: default: 32768 minimum: 301 [ 0.008128] LSM: Security Framework initializing [ 0.009062] Yama: becoming mindful. [ 0.010037] SELinux: Initializing. [ 0.011070] *** VALIDATE selinux *** [ 0.020288] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025812] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026182] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027112] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028110] *** VALIDATE tmpfs *** [ 0.030066] *** VALIDATE proc *** [ 0.031211] *** VALIDATE cgroup *** [ 0.032005] *** VALIDATE cgroup2 *** [ 0.033264] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034128] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036031] Spectre V2 : User space: Vulnerable [ 0.037012] Speculative Store Bypass: Vulnerable [ 0.040191] debug: unmapping init [mem 0xffffffffbce59000-0xffffffffbce60fff] [ 0.043000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043701] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044024] ... version: 2 [ 0.045013] ... bit width: 48 [ 0.046013] ... generic registers: 4 [ 0.047015] ... value mask: 0000ffffffffffff [ 0.048014] ... max period: 00007fffffffffff [ 0.049013] ... fixed-purpose events: 3 [ 0.050011] ... event mask: 000000070000000f [ 0.051287] rcu: Hierarchical SRCU implementation. [ 0.053478] smp: Bringing up secondary CPUs ... [ 0.054513] x86: Booting SMP configuration: [ 0.055024] .... node #0, CPUs: #1 #2 #3 [ 0.061028] smp: Brought up 1 node, 4 CPUs [ 0.063016] smpboot: Max logical packages: 1 [ 0.064013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.135325] node 0 deferred pages initialised in 68ms [ 0.142161] devtmpfs: initialized [ 0.143285] x86/mm: Memory block size: 128MB [ 0.145000] gcov: version magic: 0x41383552 [ 0.146375] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.149087] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.151284] pinctrl core: initialized pinctrl subsystem [ 0.153108] [ 0.153571] ************************************************************* [ 0.156012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.157010] ** ** [ 0.159013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.161010] ** ** [ 0.164011] ** This means that this kernel is built to expose internal ** [ 0.166012] ** IOMMU data structures, which may compromise security on ** [ 0.168025] ** your system. ** [ 0.169010] ** ** [ 0.172012] ** If you see this message and you are not debugging the ** [ 0.173018] ** kernel, report this immediately to your vendor! ** [ 0.175017] ** ** [ 0.177018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.179014] ************************************************************* [ 0.181958] NET: Registered protocol family 16 [ 0.183608] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.186086] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.188081] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.192041] cpuidle: using governor menu [ 0.194479] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.198829] PCI: Using configuration type 1 for base access [ 0.201267] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.212061] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.214066] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.218038] cryptd: max_cpu_qlen set to 1000 [ 0.220348] ACPI: Added _OSI(Module Device) [ 0.222016] ACPI: Added _OSI(Processor Device) [ 0.224026] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.226015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.231242] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.238370] ACPI: Interpreter enabled [ 0.239000] ACPI: PM: (supports S0 S3 S4 S5) [ 0.239000] ACPI: Using IOAPIC for interrupt routing [ 0.240097] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.243386] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.254038] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.256052] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.259007] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.262146] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.268834] acpiphp: Slot [2] registered [ 0.269301] acpiphp: Slot [3] registered [ 0.270192] acpiphp: Slot [4] registered [ 0.273172] acpiphp: Slot [5] registered [ 0.274223] acpiphp: Slot [6] registered [ 0.277217] acpiphp: Slot [7] registered [ 0.278150] acpiphp: Slot [8] registered [ 0.280255] acpiphp: Slot [9] registered [ 0.282157] acpiphp: Slot [10] registered [ 0.286279] acpiphp: Slot [11] registered [ 0.288136] acpiphp: Slot [12] registered [ 0.289000] acpiphp: Slot [13] registered [ 0.289000] acpiphp: Slot [14] registered [ 0.291297] acpiphp: Slot [15] registered [ 0.293159] acpiphp: Slot [16] registered [ 0.295109] acpiphp: Slot [17] registered [ 0.297213] acpiphp: Slot [18] registered [ 0.299162] acpiphp: Slot [19] registered [ 0.300141] acpiphp: Slot [20] registered [ 0.301567] acpiphp: Slot [21] registered [ 0.303143] acpiphp: Slot [22] registered [ 0.304000] acpiphp: Slot [23] registered [ 0.304000] acpiphp: Slot [24] registered [ 0.304000] acpiphp: Slot [25] registered [ 0.305013] acpiphp: Slot [26] registered [ 0.306000] acpiphp: Slot [27] registered [ 0.307151] acpiphp: Slot [28] registered [ 0.308000] acpiphp: Slot [29] registered [ 0.308000] acpiphp: Slot [30] registered [ 0.309116] acpiphp: Slot [31] registered [ 0.310000] PCI host bridge to bus 0000:00 [ 0.310000] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.312045] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.313031] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.315030] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.317033] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.319038] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.320239] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.324517] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.328502] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.336688] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.340059] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.343037] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.345023] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.346022] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.347000] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.347000] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.351071] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.354823] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.360024] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.372018] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.377025] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.383519] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.391020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.400021] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.410000] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.420267] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.427017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.432016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.454019] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.462342] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.468016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.477014] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.492022] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.518000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.525051] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.533016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.553030] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.567151] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.579024] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.590027] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.611026] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.622515] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.630019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.638018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.658019] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.675378] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.680354] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.683435] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.687431] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.690610] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.697166] iommu: Default domain type: Passthrough [ 0.699530] SCSI subsystem initialized [ 0.702183] ACPI: bus type USB registered [ 0.704125] usbcore: registered new interface driver usbfs [ 0.707097] usbcore: registered new interface driver hub [ 0.710144] usbcore: registered new device driver usb [ 0.712285] pps_core: LinuxPPS API ver. 1 registered [ 0.714016] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.719822] PTP clock support registered [ 0.723084] EDAC MC: Ver: 3.0.0 [ 0.727164] PCI: Using ACPI for IRQ routing [ 0.729947] NetLabel: Initializing [ 0.731012] NetLabel: domain hash size = 128 [ 0.733013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.736107] NetLabel: unlabeled traffic allowed by default [ 0.739040] vgaarb: loaded [ 0.741274] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.742000] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.755052] clocksource: Switched to clocksource kvm-clock [ 0.874429] VFS: Disk quotas dquot_6.6.0 [ 0.875883] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.878494] *** VALIDATE ramfs *** [ 0.879820] *** VALIDATE hugetlbfs *** [ 0.881235] pnp: PnP ACPI init [ 0.883542] pnp: PnP ACPI: found 6 devices [ 0.902912] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.906229] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.908316] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.909915] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.911677] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.913971] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.916720] NET: Registered protocol family 2 [ 0.919163] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.924592] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.930201] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.938139] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.943149] TCP: Hash tables configured (established 65536 bind 65536) [ 0.946036] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.950895] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.953846] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.957380] NET: Registered protocol family 1 [ 0.961218] RPC: Registered named UNIX socket transport module. [ 0.963803] RPC: Registered udp transport module. [ 0.965771] RPC: Registered tcp transport module. [ 0.969677] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.972383] NET: Registered protocol family 44 [ 0.974031] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.976134] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.978276] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.980168] PCI: CLS 0 bytes, default 64 [ 0.981518] Unpacking initramfs... [ 2.381654] debug: unmapping init [mem 0xffff9d23fcc54000-0xffff9d23fffbffff] [ 2.386143] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.388513] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.392682] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.979096] Initialise system trusted keyrings [ 2.981135] Key type blacklist registered [ 2.983248] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.992359] zbud: loaded [ 2.995078] *** VALIDATE nfs *** [ 2.996401] *** VALIDATE nfs4 *** [ 2.998179] pstore: using deflate compression [ 3.001863] Platform Keyring initialized [ 3.181082] NET: Registered protocol family 38 [ 3.182683] Key type asymmetric registered [ 3.183970] Asymmetric key parser 'x509' registered [ 3.185720] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.192962] io scheduler mq-deadline registered [ 3.194870] io scheduler kyber registered [ 3.196724] io scheduler bfq registered [ 3.198784] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.201868] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.204735] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.207550] ACPI: Power Button [PWRF] [ 3.303107] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.398594] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.585129] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.681843] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.879313] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.917030] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.948857] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.953404] Non-volatile memory driver v1.3 [ 3.954452] Linux agpgart interface v0.103 [ 3.988242] virtio_blk virtio1: [vda] 67984 512-byte logical blocks (34.8 MB/33.2 MiB) [ 3.990107] vda: detected capacity change from 0 to 34807808 [ 4.005904] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.010511] vdb: detected capacity change from 0 to 1073741824 [ 4.029114] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.033150] vdc: detected capacity change from 0 to 2621440000 [ 4.055704] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.058869] vdd: detected capacity change from 0 to 2621440000 [ 4.077169] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.082297] vde: detected capacity change from 0 to 4294967296 [ 4.104094] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.111174] vdf: detected capacity change from 0 to 4294967296 [ 4.125243] libphy: Fixed MDIO Bus: probed [ 4.132535] usbcore: registered new interface driver usbserial_generic [ 4.136016] usbserial: USB Serial support registered for generic [ 4.137740] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.144084] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.146288] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.149157] mousedev: PS/2 mouse device common for all mice [ 4.151976] rtc_cmos 00:05: RTC can wake from S4 [ 4.158283] rtc_cmos 00:05: registered as rtc0 [ 4.159910] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.167429] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.172673] intel_pstate: CPU model not supported [ 4.175146] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.178217] hid: raw HID events driver (C) Jiri Kosina [ 4.179592] usbcore: registered new interface driver usbhid [ 4.181167] usbhid: USB HID core driver [ 4.182233] drop_monitor: Initializing network drop monitor service [ 4.184185] Initializing XFRM netlink socket [ 4.186051] NET: Registered protocol family 10 [ 4.189424] Segment Routing with IPv6 [ 4.190519] NET: Registered protocol family 17 [ 4.192058] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.195898] mpls_gso: MPLS GSO support [ 4.205435] RAS: Correctable Errors collector initialized. [ 4.207609] AVX version of gcm_enc/dec engaged. [ 4.213124] AES CTR mode by8 optimization enabled [ 4.335119] sched_clock: Marking stable (4335101189, 0)->(5359773827, -1024672638) [ 4.339377] registered taskstats version 1 [ 4.342310] Loading compiled-in X.509 certificates [ 4.344078] zswap: loaded using pool lzo/zbud [ 4.383243] Key type big_key registered [ 4.402859] Key type encrypted registered [ 4.404428] ima: No TPM chip found, activating TPM-bypass! [ 4.409411] ima: Allocated hash algorithm: sha1 [ 4.411954] ima: No architecture policies found [ 4.414660] evm: Initialising EVM extended attributes: [ 4.417027] evm: security.selinux [ 4.420789] evm: security.ima [ 4.422032] evm: security.capability [ 4.423911] evm: HMAC attrs: 0x1 [ 4.432185] rtc_cmos 00:05: setting system clock to 2025-10-24 13:08:16 UTC (1761311296) [ 4.443176] debug: unmapping init [mem 0xffffffffbde03000-0xffffffffbdffffff] [ 4.445941] debug: unmapping init [mem 0xffffffffbcb82000-0xffffffffbce58fff] [ 4.466068] Write protecting the kernel read-only data: 28672k [ 4.472075] debug: unmapping init [mem 0xffffffffbb203000-0xffffffffbb3fffff] [ 4.476722] debug: unmapping init [mem 0xffffffffbbb14000-0xffffffffbbbfffff] [ 4.538632] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.556588] systemd[1]: Detected virtualization kvm. [ 4.561902] systemd[1]: Detected architecture x86-64. [ 4.566595] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.603849] systemd[1]: No hostname configured. [ 4.606594] systemd[1]: Set hostname to . [ 4.608294] random: systemd: uninitialized urandom read (16 bytes read) [ 4.614049] systemd[1]: Initializing machine ID from random generator. [ 4.823743] random: systemd: uninitialized urandom read (16 bytes read) [ 4.826569] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 4.832750] random: systemd: uninitialized urandom read (16 bytes read) [ 4.835081] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 4.838831] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Slices. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. [ OK ] Reached target Timers. Starting Apply Kernel Variables... Starting Journal Service... [ OK ] Reached target Initrd Root Device. Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.833140] device-mapper: uevent: version 1.0.3 [ 5.839471] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ 6.779634] random: fast init done [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 6.825149] virtio_net virtio0 ens2: renamed from eth0 [ 7.340241] scsi host0: ata_piix [ 7.519219] scsi host1: ata_piix [ 7.520418] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 7.531069] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 11.943117] random: crng init done [ 11.944673] random: 7 urandom warning(s) missed due to ratelimiting [ 12.632518] dracut-initqueue[588]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 13.638966] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 15.298933] printk: systemd: 26 output lines suppressed due to ratelimiting [ 15.662298] SELinux: Disabled at runtime. [ 15.738927] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 15.750914] systemd[1]: Detected virtualization kvm. [ 15.753507] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 16.515237] systemd[1]: initrd-switch-root.service: Succeeded. [ 16.518820] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 16.524949] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 16.527906] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 16.530683] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 16.554908] systemd[1]: Starting Journal Service... Starting Journal Service... [ 16.561593] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-getty.slice. Mounting POSIX Message Queue File System... [ OK ] Reached target rpc_pipefs.target. Mounting Kernel Debug File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Listening on Process Core Dump Socket. Mounting Huge Pages File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Starting Create list of required st…ce nodes for the current kernel... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-f[ 16.853597] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS s-binfmt_misc.automount' for details. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 17.521770] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 18.114229] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 18.125630] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 18.748212] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 18.851735] EDAC sbridge: Ver: 1.1.2 [ 21.267945] Key type dns_resolver registered [ 21.628658] NFS: Registering the id_resolver key type [ 21.630827] Key type id_resolver registered [ 21.632479] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg222-server login: [ 53.978147] hrtimer: interrupt took 2262115 ns [ 62.736881] libcfs: loading out-of-tree module taints kernel. [ 62.773110] alg: No test for adler32 (adler32-zlib) [ 63.527439] Key type ._llcrypt registered [ 63.529108] Key type .llcrypt registered [ 63.581256] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_hostid [ 75.016895] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 75.791612] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 76.169830] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 76.562636] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 76.567768] LNet: Accept secure, port 988 [ 78.218847] Key type lgssc registered [ 79.009942] Lustre: Echo OBD driver; http://www.lustre.org/ [ 89.334886] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 90.828025] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 96.769834] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 101.139027] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 106.517703] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 124.149598] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 141.056776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 141.134420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 142.329305] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 142.365203] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 142.440084] Lustre: lustre-MDT0000: new disk, initializing [ 142.545685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 142.570711] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 145.928465] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 157.167960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 157.225682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 157.278586] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 157.300163] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 157.303562] Lustre: Skipped 1 previous similar message [ 157.359464] Lustre: lustre-MDT0001: new disk, initializing [ 157.404634] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 157.437155] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 157.448286] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 160.773035] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 173.238243] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 173.384334] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 173.418925] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 173.889845] Lustre: lustre-OST0000: new disk, initializing [ 173.899954] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 173.985732] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 179.005306] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 181.297085] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 181.316429] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 192.460709] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 192.536637] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 192.542800] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 192.684425] Lustre: lustre-OST0001: new disk, initializing [ 192.700094] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 192.761608] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 196.920953] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 200.266264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 200.281949] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 206.849396] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 211.575642] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 219.398537] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing check_logdir /tmp/testlogs/ [ 223.978899] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing yml_node [ 227.993925] Lustre: DEBUG MARKER: Client: 2.15.7.13 [ 230.464363] Lustre: DEBUG MARKER: MDS: 2.15.7.13 [ 232.692519] Lustre: DEBUG MARKER: OSS: 2.15.7.13 [ 234.101475] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Oct 24 09:12:04 EDT 2025 [ 240.733642] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball 110 [ 241.864911] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 [ 254.945456] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 254.948093] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 254.960105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 256.994017] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 256.994285] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 257.005216] Lustre: Skipped 2 previous similar messages [ 257.015488] Lustre: Skipped 2 previous similar messages [ 262.114869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 262.118618] Lustre: Skipped 4 previous similar messages [ 267.233040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 267.235021] Lustre: Skipped 3 previous similar messages [ 269.280155] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 269.403923] Lustre: server umount lustre-MDT0000 complete [ 272.365640] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 272.379131] LustreError: Skipped 3 previous similar messages [ 272.887701] LustreError: 7234:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761311564 with bad export cookie 8158880777928582439 [ 272.891360] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 277.473556] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 277.475234] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 277.476969] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 277.486857] LustreError: Skipped 1 previous similar message [ 277.492087] Lustre: Skipped 1 previous similar message [ 278.562192] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 278.572644] LustreError: Skipped 1 previous similar message [ 279.337924] Lustre: server umount lustre-MDT0001 complete [ 282.544204] Lustre: server umount lustre-OST0000 complete [ 285.819795] Lustre: server umount lustre-OST0001 complete [ 292.410506] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_hostid [ 299.965798] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 307.344208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 312.916534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 317.376774] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 322.454723] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 328.351728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 328.421712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 328.621891] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 328.646752] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 328.741510] Lustre: lustre-MDT0000: new disk, initializing [ 328.844319] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 328.863178] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 333.083956] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 342.843854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 342.912335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 342.986273] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 343.024343] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 343.028090] Lustre: Skipped 1 previous similar message [ 343.100144] Lustre: lustre-MDT0001: new disk, initializing [ 343.191570] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 343.223940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 343.236227] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 346.851222] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 355.513863] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 360.819342] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 367.063885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 367.143561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 367.157225] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 367.462526] Lustre: lustre-OST0000: new disk, initializing [ 367.484929] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 367.587670] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 369.134203] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 369.149795] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 372.396977] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 382.691386] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 386.578480] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 386.832241] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 390.935406] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 391.136881] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 394.728308] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 394.737676] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 394.743873] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 394.749766] Lustre: Skipped 1 previous similar message [ 394.769826] Lustre: Skipped 4 previous similar messages [ 399.120993] Lustre: server umount lustre-OST0000 complete [ 404.964078] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 404.964497] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 404.992341] LustreError: Skipped 1 previous similar message [ 408.736740] Lustre: server umount lustre-MDT0000 complete [ 410.082538] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 410.100238] LustreError: Skipped 1 previous similar message [ 412.539419] LustreError: 14306:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761311704 with bad export cookie 8158880777928583895 [ 412.546622] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 412.558741] LustreError: 14306:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 417.965342] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 09:15:08 (1761311708) [ 420.216699] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 421.965246] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 09:15:12 (1761311712) [ 427.614749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 428.022861] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 428.037129] LustreError: Skipped 2 previous similar messages [ 428.084300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 431.965314] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 438.242899] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 438.250964] LustreError: Skipped 1 previous similar message [ 438.521095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 441.961508] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 447.576798] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 452.976794] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 459.087375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 462.432266] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 468.231356] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 488.930986] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 488.932719] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 488.941311] LustreError: Skipped 1 previous similar message [ 488.941878] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 488.941882] Lustre: Skipped 2 previous similar messages [ 488.957374] Lustre: Skipped 2 previous similar messages [ 491.184871] Lustre: server umount lustre-OST0000 complete [ 491.187036] Lustre: Skipped 1 previous similar message [ 508.384234] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 510.435568] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 510.443784] LustreError: Skipped 1 previous similar message [ 510.973847] LustreError: 18847:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761311803 with bad export cookie 8158880777928584749 [ 510.978856] LustreError: 18847:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 510.980147] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 518.675717] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 520.633329] Key type lgssc unregistered [ 520.840367] LNet: 22541:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 520.848946] LNet: Removed LNI 192.168.202.122@tcp [ 521.413940] Key type .llcrypt unregistered [ 521.416249] Key type ._llcrypt unregistered [ 540.458630] alg: No test for adler32 (adler32-zlib) [ 541.217040] Key type ._llcrypt registered [ 541.219773] Key type .llcrypt registered [ 541.350622] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 542.621141] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 543.149677] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 543.495305] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 543.499908] LNet: Accept secure, port 988 [ 545.184119] Key type lgssc registered [ 546.425735] Lustre: Echo OBD driver; http://www.lustre.org/ [ 554.144985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 555.874726] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 555.937917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 560.180559] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 561.129779] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 566.242352] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 567.555489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 567.989920] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 571.684983] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 578.692294] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 584.228849] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 590.377460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 590.714706] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 594.822127] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 601.666944] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 606.063577] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 616.419293] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 616.420838] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 616.431571] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 620.378184] Lustre: server umount lustre-OST0000 complete [ 624.103747] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 624.115649] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 624.126902] Lustre: Skipped 1 previous similar message [ 624.132431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 624.136209] Lustre: Skipped 1 previous similar message [ 626.663880] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 626.671450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 629.621890] Lustre: server umount lustre-MDT0000 complete [ 631.780689] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 632.688760] LustreError: 23986:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761311924 with bad export cookie 17523269015334052669 [ 632.689874] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 632.699314] LustreError: 23986:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 632.979479] Lustre: server umount lustre-MDT0001 complete [ 642.767857] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 645.363825] Key type lgssc unregistered [ 645.596477] LNet: 27678:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 645.603683] LNet: Removed LNI 192.168.202.122@tcp [ 646.338188] Key type .llcrypt unregistered [ 646.345650] Key type ._llcrypt unregistered [ 665.910445] alg: No test for adler32 (adler32-zlib) [ 666.671696] Key type ._llcrypt registered [ 666.673594] Key type .llcrypt registered [ 666.769472] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 667.604441] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 667.896481] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 668.071694] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 668.078672] LNet: Accept secure, port 988 [ 669.728112] Key type lgssc registered [ 670.578473] Lustre: Echo OBD driver; http://www.lustre.org/ [ 677.485302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 678.918957] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 678.980721] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 682.607745] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 684.001508] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 689.122513] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 691.335597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 691.740909] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 695.410946] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 703.033436] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 707.956965] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 712.849089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 713.174222] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 716.534613] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 722.667222] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 729.587773] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 737.761260] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 737.765116] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 737.772850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 738.503943] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.22@tcp (stopping) [ 739.498405] Lustre: server umount lustre-MDT0000 complete [ 739.814164] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 739.822270] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 742.670301] LustreError: 29122:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761312034 with bad export cookie 10409738732217037800 [ 742.673804] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 742.686340] LustreError: 29122:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 742.910892] Lustre: server umount lustre-MDT0001 complete [ 751.008464] Lustre: 28452:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761312036/real 1761312036] req@0000000036bc0d77 x1846869449783616/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761312043 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 751.024773] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 751.030548] Lustre: Skipped 1 previous similar message [ 755.169177] Lustre: 28451:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761312040/real 1761312040] req@0000000027f75099 x1846869449783680/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761312047 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 762.652111] Lustre: server umount lustre-OST0000 complete [ 776.041151] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 778.253673] Key type lgssc unregistered [ 778.431262] LNet: 33143:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 778.438192] LNet: Removed LNI 192.168.202.122@tcp [ 778.989854] Key type .llcrypt unregistered [ 778.992930] Key type ._llcrypt unregistered [ 795.555375] alg: No test for adler32 (adler32-zlib) [ 796.313381] Key type ._llcrypt registered [ 796.314968] Key type .llcrypt registered [ 796.411874] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 797.106802] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 797.467978] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 797.652789] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 797.660559] LNet: Accept secure, port 988 [ 799.304128] Key type lgssc registered [ 799.932965] Lustre: Echo OBD driver; http://www.lustre.org/ [ 805.624102] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 806.962863] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 807.014155] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 810.311965] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 812.516579] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 816.518732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 816.754469] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 819.510688] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 825.517369] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 829.691489] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 833.828684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 834.039922] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 837.017499] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 837.099139] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 843.044305] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 852.449121] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 852.454680] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 852.460799] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 855.227365] Lustre: lustre-OST0000: Not available for connect from 192.168.202.22@tcp (stopping) [ 856.548266] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 856.553094] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 856.559050] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 857.686807] Lustre: server umount lustre-OST0000 complete [ 878.049053] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 878.065510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 878.069111] Lustre: Skipped 2 previous similar messages [ 879.830911] Lustre: server umount lustre-MDT0000 complete [ 882.676075] LustreError: 37630:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761312174 with bad export cookie 1261007304743167194 [ 882.680273] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 882.683716] LustreError: 37630:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 889.008656] Lustre: server umount lustre-MDT0001 complete [ 897.403536] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 899.302079] Key type lgssc unregistered [ 899.488143] LNet: 38435:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 899.495544] LNet: Removed LNI 192.168.202.122@tcp [ 900.017391] Key type .llcrypt unregistered [ 900.020659] Key type ._llcrypt unregistered [ 917.081552] alg: No test for adler32 (adler32-zlib) [ 917.833647] Key type ._llcrypt registered [ 917.835295] Key type .llcrypt registered [ 917.927801] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 918.648581] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 918.930745] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 919.078604] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 919.084664] LNet: Accept secure, port 988 [ 920.738039] Key type lgssc registered [ 921.389144] Lustre: Echo OBD driver; http://www.lustre.org/ [ 927.578322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 928.981823] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 929.032622] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 932.230616] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 934.371497] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 938.576725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 938.839581] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 941.716363] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 946.746631] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 951.299118] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 956.657569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 956.893070] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 958.974030] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 960.378884] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 966.752165] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 974.834643] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 977.459218] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 982.003937] LNetError: 120-3: Refusing connection from 192.168.202.122 for 127.0.0.2@tcp: No matching NI [ 982.012324] LNetError: 39200:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 982.020097] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 988.558516] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2025-10-24 9h24m39s [ 989.509725] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 993.977239] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.22@tcp (stopping) [ 994.789592] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 994.793638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 994.797650] LNetError: 120-3: Refusing connection from 192.168.202.122 for 127.0.0.2@tcp: No matching NI [ 994.797733] LNetError: 39197:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 994.804020] Lustre: Skipped 2 previous similar messages [ 994.811353] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 996.851267] Lustre: server umount lustre-MDT0000 complete [ 999.082984] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 999.790487] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 999.927135] LNetError: 120-3: Refusing connection from 192.168.202.122 for 127.0.0.2@tcp: No matching NI [ 999.934448] LNetError: 39198:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 999.937957] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1004.203334] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1004.215041] LustreError: Skipped 3 previous similar messages [ 1005.029422] LNetError: 120-3: Refusing connection from 192.168.202.122 for 127.0.0.2@tcp: No matching NI [ 1005.034702] LNetError: 39199:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1005.038430] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1005.159533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1005.349842] LustreError: 11-0: MGC192.168.202.122@tcp: operation mgs_target_reg to node 0@lo failed: rc = -107 [ 1005.350222] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1005.372758] Lustre: Evicted from MGS (at 192.168.202.122@tcp) after server handle changed from 0x25dce717b838c6f6 to 0x25dce717b838cb4f [ 1005.386260] Lustre: MGC192.168.202.122@tcp: Connection restored to 192.168.202.122@tcp (at 0@lo) [ 1005.567879] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1005.617562] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:193 [ 1008.976434] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 1010.661387] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 1010.665828] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1011.408653] Lustre: DEBUG MARKER: Wait for df (28892) ... [ 1012.471671] Lustre: DEBUG MARKER: done [ 1015.266905] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1015.270880] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1015.278419] Lustre: Skipped 2 previous similar messages [ 1015.281809] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1020.907976] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1020.915925] Lustre: Skipped 2 previous similar messages [ 1021.465729] Lustre: server umount lustre-OST0000 complete [ 1026.017838] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1026.025418] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1026.031263] LustreError: Skipped 1 previous similar message [ 1026.040703] Lustre: Skipped 1 previous similar message [ 1026.041391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1026.045796] LNetError: 120-3: Refusing connection from 192.168.202.122 for 127.0.0.2@tcp: No matching NI [ 1026.074610] LNetError: 39200:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1026.081882] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1036.257102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1036.262440] Lustre: Skipped 4 previous similar messages [ 1039.328257] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1039.425901] Lustre: server umount lustre-MDT0000 complete [ 1041.380468] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1041.393540] LustreError: Skipped 4 previous similar messages [ 1042.393996] LustreError: 42592:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761312334 with bad export cookie 2728309563340671823 [ 1042.403795] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1042.422636] LustreError: 42592:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1042.727105] Lustre: server umount lustre-MDT0001 complete [ 1051.268859] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 1053.031644] Key type lgssc unregistered [ 1053.203724] LNet: 44997:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1053.212916] LNet: Removed LNI 192.168.202.122@tcp [ 1053.636480] Key type .llcrypt unregistered [ 1053.637870] Key type ._llcrypt unregistered [ 1075.232635] alg: No test for adler32 (adler32-zlib) [ 1075.986517] Key type ._llcrypt registered [ 1075.990253] Key type .llcrypt registered [ 1076.068880] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_hostid [ 1085.939647] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 1086.876940] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1087.199919] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 1087.382791] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 1087.387389] LNet: Accept secure, port 988 [ 1089.032153] Key type lgssc registered [ 1090.073843] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1098.138363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1103.262346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1108.707358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1113.435674] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1118.703960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1118.759846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1119.981364] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1120.004832] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1120.061473] Lustre: lustre-MDT0000: new disk, initializing [ 1120.124331] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1120.147692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1123.361941] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 1132.127039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1132.199936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1132.302645] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1132.359370] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1132.361919] Lustre: Skipped 1 previous similar message [ 1132.459706] Lustre: lustre-MDT0001: new disk, initializing [ 1132.541973] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1132.573270] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1132.585619] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1135.552339] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 1141.615470] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1145.300843] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1150.349264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1150.422330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1150.432640] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1150.648469] Lustre: lustre-OST0000: new disk, initializing [ 1150.651378] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1150.729199] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1154.256196] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 1160.236060] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1160.242897] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1161.851639] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1165.016052] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1165.180713] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1168.104296] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 1168.240799] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1170.404602] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1170.406162] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1170.406612] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1170.429119] Lustre: Skipped 1 previous similar message [ 1172.097480] Lustre: server umount lustre-OST0000 complete [ 1175.521307] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1175.522249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1175.529402] Lustre: Skipped 1 previous similar message [ 1175.534220] Lustre: Skipped 2 previous similar messages [ 1180.643978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1185.761105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1185.763703] Lustre: Skipped 2 previous similar messages [ 1189.348058] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1189.467215] Lustre: server umount lustre-MDT0000 complete [ 1191.906450] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1191.912725] LustreError: Skipped 1 previous similar message [ 1192.210226] LustreError: 49088:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761312484 with bad export cookie 16717478190724801547 [ 1192.214797] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1192.220348] LustreError: 49088:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1192.444334] Lustre: server umount lustre-MDT0001 complete [ 1199.100919] Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 09:28:09 (1761312489) [ 1204.988696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1205.270533] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1205.331987] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1208.571448] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 1210.338271] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1214.860278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1215.055294] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1217.972194] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 1223.224375] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1227.209596] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1232.028733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1232.241671] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1235.311670] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 1240.602576] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1249.490685] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2025-10-24 9h29m00s [ 1250.655436] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1256.131766] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1258.312307] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1258.382687] LustreError: 53638:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1258.488311] LNetError: 120-3: Refusing connection from 192.168.202.122 for 127.0.0.2@tcp: No matching NI [ 1258.493040] LNetError: 47305:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1258.502301] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1278.675896] Lustre: lustre-MDT0000: Client 7e0ff26b-dded-466d-9a75-04d85aec1db3 (at 192.168.202.22@tcp) reconnecting [ 1298.465885] LustreError: 53638:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 136 awake [ 1300.504319] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1302.902548] Lustre: DEBUG MARKER: done [ 1308.130246] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1308.136182] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1308.144950] LustreError: Skipped 2 previous similar messages [ 1308.145746] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1308.155217] Lustre: Skipped 1 previous similar message [ 1308.156946] Lustre: Skipped 2 previous similar messages [ 1313.335650] Lustre: server umount lustre-OST0000 complete [ 1317.345294] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1317.350627] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1317.360517] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1317.363527] Lustre: Skipped 1 previous similar message [ 1318.381320] LNetError: 120-3: Refusing connection from 192.168.202.122 for 127.0.0.2@tcp: No matching NI [ 1318.389851] LNetError: 47302:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1318.394956] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1323.496872] LNetError: 120-3: Refusing connection from 192.168.202.122 for 127.0.0.2@tcp: No matching NI [ 1323.507333] LNetError: 47303:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1323.522148] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1329.637056] LNetError: 120-3: Refusing connection from 192.168.202.122 for 127.0.0.2@tcp: No matching NI [ 1329.640951] LNetError: 47304:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1329.644262] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1331.169397] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1331.256374] Lustre: server umount lustre-MDT0000 complete [ 1334.202628] LustreError: 56432:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761312626 with bad export cookie 16717478190724802394 [ 1334.207060] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1334.211556] LustreError: 56432:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1342.546618] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 1344.792967] Key type lgssc unregistered [ 1344.997488] LNet: 58052:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1345.004070] LNet: Removed LNI 192.168.202.122@tcp [ 1345.661508] Key type .llcrypt unregistered [ 1345.665090] Key type ._llcrypt unregistered [ 1373.352367] alg: No test for adler32 (adler32-zlib) [ 1374.113603] Key type ._llcrypt registered [ 1374.115340] Key type .llcrypt registered [ 1374.237328] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_hostid [ 1385.129137] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 1385.737764] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1386.011149] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 1386.120779] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 1386.123470] LNet: Accept secure, port 988 [ 1387.728149] Key type lgssc registered [ 1388.413485] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1396.245537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1403.468840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1409.618077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1415.094367] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1420.503293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1420.565167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1421.725893] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1421.750061] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1421.808503] Lustre: lustre-MDT0000: new disk, initializing [ 1421.895683] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1421.909889] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1424.794427] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 1433.364795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1433.451495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1433.529684] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1433.578830] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1433.581986] Lustre: Skipped 1 previous similar message [ 1433.629713] Lustre: lustre-MDT0001: new disk, initializing [ 1433.750643] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1433.805439] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1433.822542] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1437.053521] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 1445.173917] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1450.554340] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1455.746217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1455.845228] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1455.856377] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1456.019317] Lustre: lustre-OST0000: new disk, initializing [ 1456.024086] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1456.103827] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1460.540480] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 1464.356829] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1464.368365] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1468.585605] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1472.123289] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1472.308799] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1475.595828] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 1475.770057] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1479.652944] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1479.660871] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1479.680866] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1480.169098] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1480.181612] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1480.189810] Lustre: Skipped 1 previous similar message [ 1483.403686] Lustre: server umount lustre-OST0000 complete [ 1489.889646] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1489.893470] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1489.900478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1491.671586] Lustre: server umount lustre-MDT0000 complete [ 1494.189533] LustreError: 62146:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761312786 with bad export cookie 10716723864291081327 [ 1494.192549] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1494.197932] LustreError: 62146:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1494.387224] Lustre: server umount lustre-MDT0001 complete [ 1500.643122] Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 09:33:11 (1761312791) [ 1502.013691] Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST [ 1503.401607] Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 09:33:14 (1761312794) [ 1512.691809] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 1515.905804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1516.307761] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1516.358302] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1517.352444] Lustre: Failing over lustre-MDT0000 [ 1517.364443] LustreError: 67165:0:(osp_object.c:629:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1517.378422] LustreError: 67165:0:(lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1517.382343] LustreError: 67165:0:(lod_dev.c:424:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000 get update log failed: rc = -5 [ 1523.681911] Lustre: 67217:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761312809/real 1761312809] req@0000000061d2c3ce x1846870202666112/t0(0) o251->MGC192.168.202.122@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761312815 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1523.781844] Lustre: server umount lustre-MDT0000 complete [ 1528.152874] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 09:33:38 (1761312818) [ 1533.284143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1533.544444] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1533.594226] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1536.524914] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 1539.043886] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1543.479655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1543.853651] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1547.184239] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 1553.776814] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1558.457551] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1563.718113] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1564.019600] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1567.681530] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 1574.128047] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1578.883953] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1584.616810] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1584.617340] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1584.632788] Lustre: Skipped 1 previous similar message [ 1584.634290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1584.634297] Lustre: Skipped 2 previous similar messages [ 1586.060649] Lustre: server umount lustre-MDT0000 complete [ 1589.187480] LustreError: 70530:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761312881 with bad export cookie 10716723864291082454 [ 1589.192720] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1589.204556] LustreError: 70530:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1592.534765] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1596.458397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1596.896201] Lustre: 60375:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761312881/real 1761312881] req@000000006377e0df x1846870202682624/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761312888 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1596.913736] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1596.928771] Lustre: Skipped 2 previous similar messages [ 1600.993131] Lustre: 60372:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761312886/real 1761312886] req@00000000e6564ab9 x1846870202682688/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761312893 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1602.797169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1606.178373] Lustre: Evicted from MGS (at 192.168.202.122@tcp) after server handle changed from 0x94b973932f0580d6 to 0x94b973932f058869 [ 1606.194536] Lustre: MGC192.168.202.122@tcp: Connection restored to 192.168.202.122@tcp (at 0@lo) [ 1606.400786] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1606.459946] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1609.143643] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 1613.799186] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1613.807847] LustreError: Skipped 1 previous similar message [ 1614.950385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1615.235565] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:65 [ 1616.238714] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 1618.038258] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 1622.516924] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1626.122601] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1630.689865] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1630.701780] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1630.709804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1630.712064] Lustre: Skipped 2 previous similar messages [ 1635.776917] Lustre: server umount lustre-MDT0000 complete [ 1635.778485] Lustre: Skipped 1 previous similar message [ 1636.833831] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1636.839786] LustreError: Skipped 4 previous similar messages [ 1638.108502] LustreError: 67764:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761312930 with bad export cookie 10716723864291084393 [ 1638.114719] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1638.114955] LustreError: 67764:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1642.183900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1648.143198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1649.097187] Lustre: 60372:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761312934/real 1761312934] req@0000000096e91cb2 x1846870202695936/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761312941 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1649.109638] Lustre: 60372:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1649.117257] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1649.125477] Lustre: Skipped 2 previous similar messages [ 1653.152143] Lustre: 60375:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761312938/real 1761312938] req@000000009356d488 x1846870202696128/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761312945 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1653.168065] Lustre: 60375:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1655.268404] Lustre: Evicted from MGS (at 192.168.202.122@tcp) after server handle changed from 0x94b973932f058869 to 0x94b973932f058e6d [ 1655.284068] Lustre: MGC192.168.202.122@tcp: Connection restored to (at 0@lo) [ 1655.287986] Lustre: Skipped 1 previous similar message [ 1655.471079] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1655.574386] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 1658.479396] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 1658.596374] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 1664.530242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1664.788097] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:97 [ 1666.792664] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 1667.502633] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 1672.565805] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1676.755865] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1682.405025] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1682.410922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1682.411377] Lustre: Skipped 1 previous similar message [ 1682.419645] Lustre: Skipped 5 previous similar messages [ 1683.296272] Lustre: server umount lustre-MDT0000 complete [ 1683.298379] Lustre: Skipped 1 previous similar message [ 1686.364136] LustreError: 70530:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761312978 with bad export cookie 10716723864291085933 [ 1686.378132] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1691.944561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1694.624122] Lustre: 60375:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761312979/real 1761312979] req@000000005df4ae64 x1846870202707328/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761312986 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1694.652052] Lustre: 60375:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1695.026725] Lustre: DEBUG MARKER: files compared the same [ 1710.255753] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 1712.976473] Key type lgssc unregistered [ 1713.283954] LNet: 77817:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1713.293332] LNet: Removed LNI 192.168.202.122@tcp [ 1714.035202] Key type .llcrypt unregistered [ 1714.037275] Key type ._llcrypt unregistered [ 1730.846523] alg: No test for adler32 (adler32-zlib) [ 1731.591665] Key type ._llcrypt registered [ 1731.593337] Key type .llcrypt registered [ 1731.721657] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 1732.569652] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1732.847591] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 1733.002115] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 1733.005657] LNet: Accept secure, port 988 [ 1734.672146] Key type lgssc registered [ 1735.358518] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1740.020204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1741.264453] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1741.300929] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1743.561877] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug +malloc all 8 [ 1746.401609] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1749.268548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1749.510891] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1753.826917] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug +malloc all 8 [ 1760.937390] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1766.829792] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1771.956910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1772.169235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1775.127529] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug +malloc all 8 [ 1779.369135] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:129 [ 1780.517252] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1792.685459] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 1803.233656] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1803.236054] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1803.242192] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1803.245294] Lustre: Skipped 1 previous similar message [ 1806.869147] Lustre: server umount lustre-OST0000 complete [ 1810.916434] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1810.920675] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1810.927671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1810.930386] Lustre: Skipped 1 previous similar message [ 1813.473543] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1813.474257] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1813.484603] Lustre: Skipped 1 previous similar message [ 1815.281092] Lustre: server umount lustre-MDT0000 complete [ 1817.993819] LustreError: 79258:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761313110 with bad export cookie 18326433179986907004 [ 1817.997204] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1818.002557] LustreError: 79258:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1818.247575] Lustre: server umount lustre-MDT0001 complete [ 1827.560220] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 1830.090782] Key type lgssc unregistered [ 1830.312193] LNet: 82953:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1830.319980] LNet: Removed LNI 192.168.202.122@tcp [ 1830.970537] Key type .llcrypt unregistered [ 1830.973973] Key type ._llcrypt unregistered [ 1849.936442] alg: No test for adler32 (adler32-zlib) [ 1850.696415] Key type ._llcrypt registered [ 1850.698557] Key type .llcrypt registered [ 1850.804069] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 1851.969887] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1852.246761] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 1852.486870] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 1852.491292] LNet: Accept secure, port 988 [ 1854.160084] Key type lgssc registered [ 1855.265679] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1862.126231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1879.648748] LustreError: 84389:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.122@tcp: failed processing log, type 1: rc = -5 [ 1910.368242] LustreError: 84389:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.122@tcp: failed processing log, type 4: rc = -110 [ 1940.192741] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1943.972748] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 1950.089836] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1956.835566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1957.030968] Lustre: *** cfs_fail_loc=706, val=0*** [ 1963.055050] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1963.148674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1963.213292] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:129 [ 1967.250471] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 1968.612820] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1973.730600] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1976.366886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1976.680233] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1976.777490] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:161 [ 1981.356857] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 1989.610927] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1995.062757] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1997.284550] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1997.294650] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1997.310865] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1998.817137] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1998.821503] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1998.852475] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2000.364892] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2003.256832] Lustre: server umount lustre-OST0000 complete [ 2007.522990] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2007.526509] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2007.536169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2007.540173] Lustre: Skipped 1 previous similar message [ 2010.595797] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2012.889468] Lustre: server umount lustre-MDT0000 complete [ 2015.714034] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2015.724336] LustreError: Skipped 1 previous similar message [ 2015.814143] LustreError: 84396:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761313307 with bad export cookie 5321055528356792796 [ 2015.817958] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2015.819845] LustreError: 84396:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2016.135883] Lustre: server umount lustre-MDT0001 complete [ 2024.250614] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 2026.301646] Key type lgssc unregistered [ 2026.475263] LNet: 88126:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2026.498165] LNet: Removed LNI 192.168.202.122@tcp [ 2027.057427] Key type .llcrypt unregistered [ 2027.058803] Key type ._llcrypt unregistered [ 2045.643890] alg: No test for adler32 (adler32-zlib) [ 2046.396455] Key type ._llcrypt registered [ 2046.399279] Key type .llcrypt registered [ 2046.508720] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 2047.100585] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2047.566281] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 2047.709888] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 2047.712716] LNet: Accept secure, port 988 [ 2049.368138] Key type lgssc registered [ 2050.180513] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2055.941826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2060.249335] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 2067.082918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2067.387937] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2067.509092] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2071.594424] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 2072.556263] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2072.577208] LustreError: Skipped 2 previous similar messages [ 2077.665812] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2077.690312] LustreError: Skipped 1 previous similar message [ 2078.523178] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2078.829268] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2082.832491] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 2082.942094] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2082.980182] LustreError: Skipped 4 previous similar messages [ 2082.982283] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:193 [ 2088.419390] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2088.434673] LustreError: Skipped 3 previous similar messages [ 2090.817411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2090.882060] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:161 [ 2094.238384] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 2106.339001] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2106.342569] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2106.343789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2106.347687] Lustre: Skipped 1 previous similar message [ 2106.353837] Lustre: Skipped 1 previous similar message [ 2111.454986] Lustre: server umount lustre-OST0000 complete [ 2116.577270] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2116.578621] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2116.591992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2116.595212] Lustre: Skipped 1 previous similar message [ 2120.640309] Lustre: server umount lustre-MDT0000 complete [ 2121.704026] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2121.710286] LustreError: Skipped 4 previous similar messages [ 2124.098832] Lustre: server umount lustre-MDT0001 complete [ 2133.472233] Lustre: 92993:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761313419/real 1761313419] req@0000000006aaa8fe x1846870895768064/t0(0) o251->MGC192.168.202.122@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761313425 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2133.513563] Lustre: server umount MGS complete [ 2142.213797] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 09:43:52 (1761313432) [ 2167.615080] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_hostid [ 2174.116429] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 2182.052174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2186.759598] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2191.285802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2196.124952] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2202.373364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2202.429424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2206.139417] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 2214.692789] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2214.779996] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2215.004325] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2215.040495] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 2215.117243] Lustre: lustre-MDT0001: new disk, initializing [ 2215.205881] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2215.234085] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2215.237400] LustreError: 97693:0:(lod_dev.c:424:lod_sub_recovery_thread()) lustre-MDT0001-osd get update log failed: rc = -115 [ 2219.138641] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 2227.769298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2227.827183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2227.832960] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2228.000669] Lustre: lustre-OST0000: new disk, initializing [ 2228.005180] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2228.058648] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2231.984080] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 2236.450929] Lustre: 98953:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@00000000a3192d6e x1846870895772736/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1761313540 ref 2 fl Rpc:RQU/2/0 rc 0/-115 job:'' [ 2241.632689] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2248.165898] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2248.205759] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2248.296715] Lustre: lustre-MDT0000: new disk, initializing [ 2248.443148] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2248.460789] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2249.504983] Lustre: 99546:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@0000000047d1bee6 x1846870895776192/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1761313553 ref 2 fl Rpc:RQU/2/0 rc 0/-115 job:'' [ 2250.723183] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2251.996845] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 2256.890195] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2256.902247] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2258.666401] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 2258.672804] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 2280.418732] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2280.427857] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2280.430275] Lustre: Skipped 2 previous similar messages [ 2280.440069] Lustre: Skipped 1 previous similar message [ 2285.541299] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2285.548866] Lustre: Skipped 1 previous similar message [ 2285.929539] Lustre: server umount lustre-OST0000 complete [ 2290.660208] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2290.660368] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2290.661720] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2290.671969] Lustre: Skipped 1 previous similar message [ 2295.233272] Lustre: server umount lustre-MDT0000 complete [ 2295.822175] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2295.829760] LustreError: Skipped 3 previous similar messages [ 2314.720418] Lustre: 100741:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761313600/real 1761313600] req@00000000f12ce131 x1846870895786880/t0(0) o251->MGC192.168.202.122@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761313606 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2314.750754] Lustre: server umount MGS complete [ 2314.753694] Lustre: Skipped 1 previous similar message [ 2325.596492] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 09:46:56 (1761313616) [ 2341.398671] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 2344.030337] Key type lgssc unregistered [ 2344.306284] LNet: 102147:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2344.337964] LNet: Removed LNI 192.168.202.122@tcp [ 2345.110122] Key type .llcrypt unregistered [ 2345.111729] Key type ._llcrypt unregistered [ 2363.216988] alg: No test for adler32 (adler32-zlib) [ 2363.972631] Key type ._llcrypt registered [ 2363.974569] Key type .llcrypt registered [ 2364.143755] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 2366.144460] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2366.552550] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 2366.711262] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 2366.715124] LNet: Accept secure, port 988 [ 2368.344192] Key type lgssc registered [ 2369.506722] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2376.782608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2378.184787] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2378.250737] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2382.057365] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 2383.331979] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2388.450957] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2390.261526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2390.567797] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2394.591992] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 2401.143095] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2406.153051] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2411.340633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2411.527451] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2414.313287] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 2418.658868] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2428.982718] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 2434.019784] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2434.019956] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2434.020944] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2434.037340] Lustre: Skipped 1 previous similar message [ 2435.658559] Lustre: server umount lustre-OST0000 complete [ 2439.140050] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2439.154089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2439.156903] Lustre: Skipped 2 previous similar messages [ 2444.262146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2444.273791] Lustre: Skipped 1 previous similar message [ 2444.512146] Lustre: server umount lustre-MDT0000 complete [ 2448.058881] LustreError: 103418:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761313740 with bad export cookie 14245224071238478573 [ 2448.065313] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2448.077056] LustreError: 103418:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2448.488785] Lustre: server umount lustre-MDT0001 complete [ 2458.943202] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 2461.792892] Key type lgssc unregistered [ 2462.084863] LNet: 107111:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2462.102107] LNet: Removed LNI 192.168.202.122@tcp [ 2462.865646] Key type .llcrypt unregistered [ 2462.872350] Key type ._llcrypt unregistered [ 2475.625390] alg: No test for adler32 (adler32-zlib) [ 2476.381708] Key type ._llcrypt registered [ 2476.389597] Key type .llcrypt registered [ 2476.482088] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 2477.085865] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2477.410268] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 2477.564235] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 2477.566651] LNet: Accept secure, port 988 [ 2479.176129] Key type lgssc registered [ 2480.181876] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2485.452860] LustreError: 108240:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 716 sleeping [ 2485.479890] LustreError: 108241:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 716 waking [ 2485.483433] LustreError: 108240:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 716 awake: rc=4978 [ 2485.499716] LustreError: 108240:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2485.525525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2485.526395] LustreError: 108240:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2487.221342] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2487.285300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2492.385268] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2495.981588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2496.222462] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2500.324930] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 2506.046907] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing lsmod [ 2509.134668] LustreError: 109539:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 716 sleeping [ 2509.227339] LustreError: 109550:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 716 waking [ 2509.240465] LustreError: 109539:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 716 awake: rc=4913 [ 2509.248959] LustreError: 109539:0:(genops.c:522:class_register_device()) lustre-OST0000-osd: already exists, won't add [ 2509.260358] LustreError: 109539:0:(obd_mount.c:195:lustre_start_simple()) lustre-OST0000-osd attach error -17 [ 2509.273257] LustreError: 109539:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -17 [ 2509.275490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2509.287104] LustreError: 109539:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -17 [ 2509.883361] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2516.961427] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2516.993608] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2517.000573] Lustre: Skipped 1 previous similar message [ 2522.082828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2522.090795] Lustre: Skipped 3 previous similar messages [ 2527.203607] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2527.210039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2527.223313] Lustre: Skipped 1 previous similar message [ 2527.458799] Lustre: server umount lustre-MDT0000 complete [ 2531.117696] LustreError: 108259:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761313823 with bad export cookie 7027164468149098232 [ 2531.137568] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2531.513513] Lustre: server umount lustre-MDT0001 complete [ 2540.001270] Lustre: 110138:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761313826/real 1761313826] req@00000000820af146 x1846871346653760/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761313832 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2540.303524] Lustre: server umount lustre-OST0000 complete [ 2547.939664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2548.469914] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2548.576742] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2552.136665] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 2553.827328] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2558.948942] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2559.545292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2559.856809] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2563.495326] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 2570.833398] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2577.644319] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2585.466032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2585.902766] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2591.127867] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 2598.933369] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2599.216389] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:65 [ 2604.521910] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2604.539257] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2604.541932] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2604.544204] Lustre: Skipped 3 previous similar messages [ 2604.547927] Lustre: Skipped 1 previous similar message [ 2606.476916] Lustre: server umount lustre-OST0000 complete [ 2611.169410] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2611.172815] LustreError: Skipped 1 previous similar message [ 2611.193443] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2611.211433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2614.758702] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2616.166752] Lustre: server umount lustre-MDT0000 complete [ 2619.641065] LustreError: 110596:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761313911 with bad export cookie 7027164468149098925 [ 2619.647463] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2619.655182] LustreError: 110596:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 2630.908957] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 2633.390513] Key type lgssc unregistered [ 2633.634740] LNet: 114272:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2633.647717] LNet: Removed LNI 192.168.202.122@tcp [ 2634.389048] Key type .llcrypt unregistered [ 2634.393143] Key type ._llcrypt unregistered [ 2653.631035] alg: No test for adler32 (adler32-zlib) [ 2654.374435] Key type ._llcrypt registered [ 2654.376103] Key type .llcrypt registered [ 2654.481792] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 2655.354486] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2655.874682] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 2656.108927] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 2656.113426] LNet: Accept secure, port 988 [ 2657.784645] Key type lgssc registered [ 2658.665447] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2665.194673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2666.648320] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2666.714583] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2670.331521] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 2672.101699] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2677.035651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2677.224576] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2677.326434] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2681.483431] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 2687.758449] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2693.034747] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2699.719847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2700.031366] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2704.185929] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 2711.319706] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2718.456097] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 2725.704148] Lustre: Setting parameter lustre-client.llite.some_wrong_param in log lustre-client [ 2728.815254] Lustre: Modifying parameter lustre-client.llite.some_wrong_param in log lustre-client [ 2734.061187] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2734.062444] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2734.074530] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2734.081158] LustreError: Skipped 1 previous similar message [ 2738.827948] Lustre: server umount lustre-OST0000 complete [ 2743.780085] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2743.785968] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2743.790789] Lustre: Skipped 1 previous similar message [ 2743.794206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2743.797691] Lustre: Skipped 1 previous similar message [ 2748.565179] Lustre: server umount lustre-MDT0000 complete [ 2749.415292] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2749.438310] LustreError: Skipped 1 previous similar message [ 2752.240049] LustreError: 118358:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761314044 with bad export cookie 2757734938911585158 [ 2752.246812] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2752.604140] Lustre: server umount lustre-MDT0001 complete [ 2762.879969] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 2765.359392] Key type lgssc unregistered [ 2765.695964] LNet: 119515:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2765.707202] LNet: Removed LNI 192.168.202.122@tcp [ 2766.562627] Key type .llcrypt unregistered [ 2766.570985] Key type ._llcrypt unregistered [ 2780.294439] alg: No test for adler32 (adler32-zlib) [ 2781.049357] Key type ._llcrypt registered [ 2781.050923] Key type .llcrypt registered [ 2781.135652] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 2781.827573] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2782.058613] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 2782.204770] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 2782.208731] LNet: Accept secure, port 988 [ 2783.859800] Key type lgssc registered [ 2784.832814] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2791.999663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2793.656214] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2793.757633] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2798.965642] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 2799.080658] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2804.210526] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2807.728630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2808.117190] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2812.084909] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 2819.274838] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2824.823599] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2831.233733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2831.546961] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2835.480830] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 2842.317492] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2844.854756] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 2859.489764] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2859.492609] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2859.523722] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2860.513177] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2860.518656] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2860.563938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2861.866787] Lustre: server umount lustre-OST0000 complete [ 2869.730984] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2869.733741] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2869.747096] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2869.754357] Lustre: Skipped 1 previous similar message [ 2872.040923] Lustre: server umount lustre-MDT0000 complete [ 2875.876686] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2875.892342] LustreError: Skipped 1 previous similar message [ 2876.262906] LustreError: 120765:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761314168 with bad export cookie 3822893701834142896 [ 2876.273777] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2876.283373] LustreError: 120765:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2876.646436] Lustre: server umount lustre-MDT0001 complete [ 2888.238922] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 2890.768449] Key type lgssc unregistered [ 2891.059735] LNet: 124458:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2891.068717] LNet: Removed LNI 192.168.202.122@tcp [ 2891.903994] Key type .llcrypt unregistered [ 2891.905752] Key type ._llcrypt unregistered [ 2913.052965] alg: No test for adler32 (adler32-zlib) [ 2913.806600] Key type ._llcrypt registered [ 2913.807992] Key type .llcrypt registered [ 2913.895538] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 2914.548115] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2914.726544] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 2914.833218] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 2914.835641] LNet: Accept secure, port 988 [ 2916.456183] Key type lgssc registered [ 2917.335335] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2925.086848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2926.611938] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2926.663597] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2931.552384] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 2931.685484] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2936.809868] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2940.445340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2940.890265] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2944.885574] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 2953.024880] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2960.017637] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2967.544503] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2967.974474] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2973.560759] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 2976.181507] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 2982.264553] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2989.309884] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash in log lustre-MDT0000 [ 2993.413549] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids in log lustre-MDT0000 [ 2993.416944] Lustre: Skipped 2 previous similar messages [ 2998.039060] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash in log lustre-MDT0000 [ 2998.042474] Lustre: Skipped 2 previous similar messages [ 2998.852872] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 2998.901066] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3006.926259] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids in log lustre-MDT0000 [ 3006.932899] Lustre: Skipped 2 previous similar messages [ 3014.216830] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.22@tcp 192.168.0.[2,10]@tcp [ 3020.790878] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3020.807735] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3020.843858] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3022.306223] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3022.307062] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3022.307557] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3022.607114] Lustre: server umount lustre-OST0000 complete [ 3027.429815] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3027.446359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3027.455431] Lustre: Skipped 2 previous similar messages [ 3032.544890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3032.764441] Lustre: server umount lustre-MDT0000 complete [ 3036.866538] LustreError: 129229:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761314328 with bad export cookie 54908076470500890 [ 3036.868875] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3036.872071] LustreError: 129229:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3037.262719] Lustre: server umount lustre-MDT0001 complete [ 3048.365907] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 3050.860045] Key type lgssc unregistered [ 3051.146791] LNet: 130585:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3051.160955] LNet: Removed LNI 192.168.202.122@tcp [ 3051.995041] Key type .llcrypt unregistered [ 3051.996711] Key type ._llcrypt unregistered [ 3079.196398] alg: No test for adler32 (adler32-zlib) [ 3079.952444] Key type ._llcrypt registered [ 3079.954572] Key type .llcrypt registered [ 3080.057078] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 3080.860350] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3081.475465] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 3081.640649] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 3081.646541] LNet: Accept secure, port 988 [ 3083.297497] Key type lgssc registered [ 3084.175718] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3091.256871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3092.710968] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3092.763570] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3092.797385] Lustre: lustre-MDT0000: nosquash_nids set to 2@gni 0@lo 192.168.202.22@tcp 192.168.0.[2,10]@tcp [ 3092.814502] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3096.374314] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 3098.083328] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3103.132638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3103.203823] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3103.396210] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3103.412929] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.22@tcp 192.168.0.[2,10]@tcp [ 3103.448331] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3106.994169] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 3113.826713] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3119.415360] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3125.688538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3126.053072] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3130.342745] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 3138.072447] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3143.487660] Lustre: lustre-OST0000: deleting orphan objects from 0x0:167 to 0x0:193 [ 3153.890077] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3153.893746] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3153.904407] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3153.904407] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3156.378362] Lustre: server umount lustre-OST0000 complete [ 3160.034102] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3160.041297] LustreError: Skipped 1 previous similar message [ 3160.045578] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3160.060927] Lustre: Skipped 1 previous similar message [ 3160.070458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3164.131331] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3164.132395] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3165.850931] Lustre: server umount lustre-MDT0000 complete [ 3169.255686] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3169.275235] LustreError: Skipped 1 previous similar message [ 3169.816053] LustreError: 132196:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761314461 with bad export cookie 4678070191789090837 [ 3169.820166] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3169.828625] LustreError: 132196:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3170.305018] Lustre: server umount lustre-MDT0001 complete [ 3181.814811] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 3185.172062] Key type lgssc unregistered [ 3185.504600] LNet: 135936:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3185.515961] LNet: Removed LNI 192.168.202.122@tcp [ 3186.467548] Key type .llcrypt unregistered [ 3186.469126] Key type ._llcrypt unregistered [ 3210.908828] alg: No test for adler32 (adler32-zlib) [ 3211.661386] Key type ._llcrypt registered [ 3211.672868] Key type .llcrypt registered [ 3211.777268] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_hostid [ 3224.886383] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 3226.078961] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3226.715121] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 3227.044772] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 3227.059141] LNet: Accept secure, port 988 [ 3228.808094] Key type lgssc registered [ 3229.728649] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3238.988710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3247.383471] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3254.303121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3260.559174] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3267.043540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3267.118800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3268.415532] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3268.458147] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3268.537731] Lustre: lustre-MDT0000: new disk, initializing [ 3268.683535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3268.721868] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3273.503824] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 3285.807349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3285.888706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3285.977629] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3286.030794] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3286.036161] Lustre: Skipped 1 previous similar message [ 3286.112534] Lustre: lustre-MDT0001: new disk, initializing [ 3286.239233] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3286.275902] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3286.289036] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3290.872629] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 3301.180311] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3306.620955] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3311.685390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3311.724451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3311.729371] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3311.885577] Lustre: lustre-OST0000: new disk, initializing [ 3311.889476] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3311.959862] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3314.783295] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 3319.350604] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3319.367855] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3322.290255] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3325.364079] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3325.512166] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3328.786111] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3328.948878] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3335.142144] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3335.143452] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3335.145776] LustreError: Skipped 1 previous similar message [ 3335.149607] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3335.162101] Lustre: Skipped 1 previous similar message [ 3335.182446] Lustre: Skipped 1 previous similar message [ 3336.870750] Lustre: server umount lustre-OST0000 complete [ 3342.821423] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3342.825852] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3342.834907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3345.380894] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3345.384425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3345.394523] Lustre: Skipped 1 previous similar message [ 3346.659954] Lustre: server umount lustre-MDT0000 complete [ 3349.953962] LustreError: 139458:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761314642 with bad export cookie 11416793503390544904 [ 3349.963066] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3350.381965] Lustre: server umount lustre-MDT0001 complete [ 3358.710084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3359.095791] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3359.186090] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3362.942847] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 3364.324064] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3369.451473] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3371.184976] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3371.429494] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3375.181773] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 3382.604577] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3388.339131] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3395.083877] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3395.348227] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3399.796759] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 3407.190509] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3411.466728] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3411.649375] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3415.700839] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3415.885937] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3423.757889] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3423.819448] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3423.823570] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 3423.943492] Lustre: lustre-OST0001: new disk, initializing [ 3423.945525] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 3424.035376] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3427.929364] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 3431.027227] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3431.064859] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3435.752569] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 3435.920827] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3439.303505] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 3439.507529] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3445.478753] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff992348e13800.ost_server_uuid 40 [ 3447.053877] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff992348e13800.ost_server_uuid in FULL state after 0 sec [ 3456.994034] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3456.994398] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3457.022389] LustreError: Skipped 1 previous similar message [ 3457.023155] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3457.088272] Lustre: Skipped 1 previous similar message [ 3457.108193] Lustre: Skipped 1 previous similar message [ 3465.186140] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3465.204755] Lustre: Skipped 3 previous similar messages [ 3468.257744] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3468.353619] Lustre: server umount lustre-OST0001 complete [ 3470.311079] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3470.319814] LustreError: Skipped 1 previous similar message [ 3473.382362] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3473.395372] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3473.403093] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3475.428215] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3475.436827] LustreError: Skipped 1 previous similar message [ 3485.674295] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3485.691846] LustreError: Skipped 3 previous similar messages [ 3486.177236] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3486.425676] Lustre: server umount lustre-OST0000 complete [ 3490.786900] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3490.793375] Lustre: Skipped 1 previous similar message [ 3490.798490] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3490.803319] Lustre: Skipped 5 previous similar messages [ 3496.022727] Lustre: server umount lustre-MDT0000 complete [ 3499.716583] LustreError: 143804:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761314791 with bad export cookie 11416793503390545758 [ 3499.719927] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3499.739352] LustreError: 143804:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 3519.529984] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 3522.188850] Key type lgssc unregistered [ 3522.473989] LNet: 150476:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3522.481584] LNet: Removed LNI 192.168.202.122@tcp [ 3523.246483] Key type .llcrypt unregistered [ 3523.252144] Key type ._llcrypt unregistered [ 3553.536421] alg: No test for adler32 (adler32-zlib) [ 3554.290391] Key type ._llcrypt registered [ 3554.292470] Key type .llcrypt registered [ 3554.367787] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_hostid [ 3566.694573] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 3567.759812] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3568.066933] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 3568.295862] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 3568.300493] LNet: Accept secure, port 988 [ 3569.944142] Key type lgssc registered [ 3570.938793] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3578.946200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3585.416718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3591.289716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3597.588547] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3604.426536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3604.538954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3605.740446] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3605.778726] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3605.873205] Lustre: lustre-MDT0000: new disk, initializing [ 3605.991259] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3606.017985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3610.189568] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 3620.165938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3620.220450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3620.293882] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3620.342578] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3620.346621] Lustre: Skipped 1 previous similar message [ 3620.425861] Lustre: lustre-MDT0001: new disk, initializing [ 3620.537237] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3620.595486] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3620.608156] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3625.132311] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 3633.793501] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3640.245639] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3647.038178] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3647.119807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3647.124461] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3647.302501] Lustre: lustre-OST0000: new disk, initializing [ 3647.307868] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3647.383938] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3651.263842] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 3653.159104] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3653.166557] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3660.277260] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3664.935992] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3665.112574] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3668.679955] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3668.968981] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3673.575294] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3673.579132] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3673.614891] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3673.624435] Lustre: Skipped 1 previous similar message [ 3678.694127] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3678.697498] Lustre: Skipped 1 previous similar message [ 3683.811108] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3683.813206] Lustre: Skipped 1 previous similar message [ 3685.344929] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3685.575333] Lustre: server umount lustre-OST0000 complete [ 3692.513680] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3692.523830] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3692.534950] Lustre: Skipped 1 previous similar message [ 3692.542813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3694.053582] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3695.272296] Lustre: server umount lustre-MDT0000 complete [ 3698.572168] LustreError: 154570:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761314990 with bad export cookie 6709457392968658818 [ 3698.588760] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3698.956612] Lustre: server umount lustre-MDT0001 complete [ 3707.587756] Lustre: DEBUG MARKER: == conf-sanity test 47: server restart does not make client loss lru_resize settings ========================================================== 10:09:57 (1761314997) [ 3722.683774] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_hostid [ 3730.652565] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 3739.950659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3747.261448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3753.084608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3759.946045] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3766.820690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3766.866213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3767.031371] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3767.060616] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3767.189788] Lustre: lustre-MDT0000: new disk, initializing [ 3767.321108] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3767.332884] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3771.288856] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 3781.517185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3781.575994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3781.640191] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3781.677141] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3781.680284] Lustre: Skipped 1 previous similar message [ 3781.735982] Lustre: lustre-MDT0001: new disk, initializing [ 3781.817755] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3781.855705] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3781.871819] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3785.859930] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 3795.392279] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3800.558651] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3807.040297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3807.124233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3807.128976] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3807.320827] Lustre: lustre-OST0000: new disk, initializing [ 3807.323521] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3807.420768] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3811.475667] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 3813.457267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3813.475560] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3820.850204] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3825.218195] Lustre: Failing over lustre-OST0000 [ 3825.524556] Lustre: server umount lustre-OST0000 complete [ 3834.272159] Lustre: 152796:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761315119/real 1761315119] req@00000000c6d7acff x1846872490675200/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1761315126 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 3834.299523] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3836.384161] Lustre: 152795:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761315121/real 1761315121] req@00000000ff20f2bc x1846872490675328/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1761315128 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 3836.415213] Lustre: 152795:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3841.504309] Lustre: 152795:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761315126/real 1761315126] req@00000000ac4d9ec7 x1846872490676160/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761315133 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3844.297820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3844.716694] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3844.739244] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3848.836240] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 3851.881906] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3852.222850] Lustre: Failing over lustre-MDT0000 [ 3852.239870] LustreError: 152793:0:(import.c:692:ptlrpc_connect_import_locked()) can't connect to a closed import [ 3852.350765] Lustre: server umount lustre-MDT0000 complete [ 3852.497160] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3852.969028] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3852.981702] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 3852.983272] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.202.122@tcp (at 0@lo) [ 3852.985546] Lustre: Skipped 2 previous similar messages [ 3857.579263] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3857.600762] LustreError: Skipped 3 previous similar messages [ 3859.669413] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3859.689347] LustreError: Skipped 3 previous similar messages [ 3863.525135] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3863.539151] LustreError: Skipped 2 previous similar messages [ 3865.579284] Lustre: 152794:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761315150/real 1761315150] req@00000000c0b3a6f3 x1846872490682880/t0(0) o400->MGC192.168.202.122@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761315157 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3865.617987] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3870.691814] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3870.705310] LustreError: Skipped 7 previous similar messages [ 3871.148337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3871.725860] Lustre: Evicted from MGS (at 192.168.202.122@tcp) after server handle changed from 0x5d1cc7f35f83f2d1 to 0x5d1cc7f35f83f793 [ 3871.743340] Lustre: MGC192.168.202.122@tcp: Connection restored to 192.168.202.122@tcp (at 0@lo) [ 3871.979326] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3872.089273] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3875.503237] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 3876.009752] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3877.360490] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 3877.389868] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 3877.442321] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 3882.983144] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3882.986974] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3882.999943] Lustre: Skipped 1 previous similar message [ 3883.006553] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3883.009642] Lustre: Skipped 2 previous similar messages [ 3892.707911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3892.718309] Lustre: Skipped 3 previous similar messages [ 3897.313523] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3897.522706] Lustre: server umount lustre-OST0000 complete [ 3902.949308] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3902.951479] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3902.967512] LustreError: Skipped 1 previous similar message [ 3902.980592] Lustre: Skipped 2 previous similar messages [ 3913.185254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3913.188270] Lustre: Skipped 6 previous similar messages [ 3915.233872] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3915.388070] Lustre: server umount lustre-MDT0000 complete [ 3918.305873] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3918.326390] LustreError: Skipped 5 previous similar messages [ 3919.375667] LustreError: 161562:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761315211 with bad export cookie 6709457392968660883 [ 3919.380181] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3919.406822] LustreError: 161562:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 3931.601095] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 3934.660463] Key type lgssc unregistered [ 3934.887670] LNet: 167433:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3934.903773] LNet: Removed LNI 192.168.202.122@tcp [ 3935.910448] Key type .llcrypt unregistered [ 3935.920854] Key type ._llcrypt unregistered [ 3959.688439] alg: No test for adler32 (adler32-zlib) [ 3960.446978] Key type ._llcrypt registered [ 3960.452098] Key type .llcrypt registered [ 3960.561417] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 3961.471986] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3961.965280] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 3962.180884] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 3962.187734] LNet: Accept secure, port 988 [ 3963.840132] Key type lgssc registered [ 3965.014709] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3972.607918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3974.015397] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3974.078522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3978.672082] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 3979.238078] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3984.356741] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3988.354538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3988.699082] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3994.079477] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 4003.539249] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4009.915523] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4016.117920] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4016.352409] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4020.518418] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 4024.559517] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:65 [ 4029.273747] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4048.626733] Lustre: lustre-MDT0000: Client 8eb96965-430c-475c-a0d1-4c59cd69cdb6 (at 192.168.202.22@tcp) reconnecting [ 4057.793618] Lustre: lustre-MDT0000: Client 8eb96965-430c-475c-a0d1-4c59cd69cdb6 (at 192.168.202.22@tcp) reconnecting [ 4074.724450] Lustre: lustre-MDT0000: Client 8eb96965-430c-475c-a0d1-4c59cd69cdb6 (at 192.168.202.22@tcp) reconnecting [ 4131.554580] Lustre: lustre-MDT0000: Client 8eb96965-430c-475c-a0d1-4c59cd69cdb6 (at 192.168.202.22@tcp) reconnecting [ 4184.821341] Lustre: lustre-MDT0000: Client 8eb96965-430c-475c-a0d1-4c59cd69cdb6 (at 192.168.202.22@tcp) reconnecting [ 4233.980808] Lustre: lustre-MDT0000: Client 8eb96965-430c-475c-a0d1-4c59cd69cdb6 (at 192.168.202.22@tcp) reconnecting [ 4283.068058] Lustre: lustre-MDT0000: Client 8eb96965-430c-475c-a0d1-4c59cd69cdb6 (at 192.168.202.22@tcp) reconnecting [ 4328.151758] Lustre: lustre-MDT0000: Client 8eb96965-430c-475c-a0d1-4c59cd69cdb6 (at 192.168.202.22@tcp) reconnecting [ 4455.114643] Lustre: lustre-MDT0000: Client 8eb96965-430c-475c-a0d1-4c59cd69cdb6 (at 192.168.202.22@tcp) reconnecting [ 4455.118754] Lustre: Skipped 1 previous similar message [ 4590.334388] Lustre: lustre-MDT0000: Client 8eb96965-430c-475c-a0d1-4c59cd69cdb6 (at 192.168.202.22@tcp) reconnecting [ 4590.342512] Lustre: Skipped 1 previous similar message [ 4885.245297] Lustre: lustre-MDT0000: Client 8eb96965-430c-475c-a0d1-4c59cd69cdb6 (at 192.168.202.22@tcp) reconnecting [ 4885.253823] Lustre: Skipped 11 previous similar messages [ 5277.070238] Lustre: Failing over lustre-MDT0000 [ 5277.155863] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5277.158983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5277.167605] Lustre: Skipped 2 previous similar messages [ 5277.180753] Lustre: Skipped 2 previous similar messages [ 5277.304030] Lustre: server umount lustre-MDT0000 complete [ 5277.881119] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5282.279540] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5282.295511] LustreError: Skipped 2 previous similar messages [ 5285.045929] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5285.057536] LustreError: Skipped 1 previous similar message [ 5287.397649] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5287.407089] LustreError: Skipped 2 previous similar messages [ 5289.440168] Lustre: 168205:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761316574/real 1761316574] req@00000000e7d2f9d7 x1846872904275328/t0(0) o400->MGC192.168.202.122@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761316581 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 5289.461503] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5294.565138] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5294.571237] LustreError: Skipped 6 previous similar messages [ 5295.078745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5295.587326] Lustre: Evicted from MGS (at 192.168.202.122@tcp) after server handle changed from 0xf9db4355e7f6d44e to 0xf9db4355e7f94076 [ 5295.596286] Lustre: MGC192.168.202.122@tcp: Connection restored to 192.168.202.122@tcp (at 0@lo) [ 5295.743358] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5295.779246] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5298.814912] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 5300.392547] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5301.221321] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 5301.238018] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 5301.319059] Lustre: lustre-OST0000: deleting orphan objects from 0x0:68 to 0x0:97 [ 5306.637431] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5308.256821] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5326.826140] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5326.829429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5326.830256] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5326.837925] Lustre: Skipped 2 previous similar messages [ 5326.840934] Lustre: Skipped 1 previous similar message [ 5329.650102] Lustre: server umount lustre-MDT0000 complete [ 5331.940606] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5331.955280] LustreError: Skipped 6 previous similar messages [ 5332.794216] LustreError: 168875:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761316624 with bad export cookie 18004057971586842742 [ 5332.798371] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5332.806137] LustreError: 168875:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5337.058463] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5337.069193] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5337.072468] Lustre: Skipped 1 previous similar message [ 5343.203285] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5343.207710] Lustre: Skipped 1 previous similar message [ 5347.299771] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 5347.492712] Lustre: server umount lustre-MDT0001 complete [ 5364.192147] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 5364.436820] Lustre: server umount lustre-OST0000 complete [ 5373.722364] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_hostid [ 5381.684676] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 5389.595540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5395.237913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5400.676676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5406.620901] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5412.976823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5413.045958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5413.228166] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5413.261627] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5413.364042] Lustre: lustre-MDT0000: new disk, initializing [ 5413.434831] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5413.453213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5417.246078] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 5426.051350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5426.092041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5426.144379] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5426.184184] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 5426.187758] Lustre: Skipped 1 previous similar message [ 5426.271925] Lustre: lustre-MDT0001: new disk, initializing [ 5426.354245] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5426.387345] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5426.396310] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5429.667764] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 5438.787841] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5445.767482] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5452.196359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5452.244837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5452.250351] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 5452.455685] Lustre: lustre-OST0000: new disk, initializing [ 5452.461680] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5452.539098] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5455.827764] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 5460.512186] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5460.518566] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5464.147318] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5467.142147] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 5467.274740] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5470.393759] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 5470.561430] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5475.808814] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5475.812232] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5475.820080] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5475.822168] Lustre: Skipped 1 previous similar message [ 5478.314927] Lustre: server umount lustre-OST0000 complete [ 5482.979422] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5482.990721] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5486.051364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5486.057920] Lustre: Skipped 3 previous similar messages [ 5487.401310] Lustre: server umount lustre-MDT0000 complete [ 5490.196714] LustreError: 178024:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761316782 with bad export cookie 18004057971586843463 [ 5490.206848] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5490.211329] LustreError: 178024:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5496.726777] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 10:39:47 (1761316787) [ 5507.954152] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_hostid [ 5515.586635] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 5524.576788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5530.089965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5535.436601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5541.072828] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5547.590902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5547.698431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5548.029996] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5548.052299] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5548.154517] Lustre: lustre-MDT0000: new disk, initializing [ 5548.298347] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5548.329340] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5552.773102] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 5562.828852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5562.886530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5562.963631] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5563.002227] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 5563.005858] Lustre: Skipped 1 previous similar message [ 5563.061238] Lustre: lustre-MDT0001: new disk, initializing [ 5563.124498] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5563.161320] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5563.167350] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5567.587591] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 5578.386246] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5584.619487] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5591.048567] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5591.126674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5591.132484] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 5591.300860] Lustre: lustre-OST0000: new disk, initializing [ 5591.305563] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5591.360977] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5595.495790] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 5596.732743] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5596.739494] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5604.618571] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5612.513098] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5612.518309] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5612.533531] LustreError: Skipped 1 previous similar message [ 5612.544473] Lustre: Skipped 2 previous similar messages [ 5612.547794] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5618.773496] Lustre: server umount lustre-OST0000 complete [ 5618.775928] Lustre: Skipped 1 previous similar message [ 5624.804228] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5632.298468] LustreError: 182916:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761316924 with bad export cookie 18004057971586844324 [ 5632.304977] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5632.314856] LustreError: 182916:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5639.756652] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 10:42:10 (1761316930) [ 5654.070296] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_hostid [ 5660.437765] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 5669.651288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5677.110167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5685.190113] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5692.271301] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5701.450375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5701.544508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5701.928165] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5701.972689] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5702.179147] Lustre: lustre-MDT0000: new disk, initializing [ 5702.337501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5702.391845] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5707.312468] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 5718.128592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5718.202774] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5718.341928] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5718.734306] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5723.129630] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 5732.886277] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5738.274658] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5744.321840] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5744.364588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5744.377740] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 5744.534810] Lustre: lustre-OST0000: new disk, initializing [ 5744.537814] Lustre: Skipped 1 previous similar message [ 5744.555429] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5744.558753] Lustre: Skipped 2 previous similar messages [ 5749.882444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5749.887683] Lustre: Skipped 1 previous similar message [ 5749.891098] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5750.150442] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 5760.366629] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5767.141617] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5767.155111] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5767.179936] Lustre: Skipped 2 previous similar messages [ 5767.186631] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5767.190785] Lustre: Skipped 6 previous similar messages [ 5768.800152] Lustre: server umount lustre-OST0000 complete [ 5768.802094] Lustre: Skipped 2 previous similar messages [ 5777.376407] Lustre: 168208:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761317062/real 1761317062] req@00000000afb332d1 x1846872904341120/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1761317069 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 5781.461921] LustreError: 189382:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761317073 with bad export cookie 18004057971586845409 [ 5781.465183] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5781.468476] LustreError: 189382:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5791.119158] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 5793.482319] Key type lgssc unregistered [ 5793.745987] LNet: 193346:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5793.762664] LNet: Removed LNI 192.168.202.122@tcp [ 5794.457189] Key type .llcrypt unregistered [ 5794.460197] Key type ._llcrypt unregistered [ 5812.489363] alg: No test for adler32 (adler32-zlib) [ 5813.249204] Key type ._llcrypt registered [ 5813.255630] Key type .llcrypt registered [ 5813.388245] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 5814.230850] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5814.566208] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 5814.731375] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 5814.734699] LNet: Accept secure, port 988 [ 5816.392157] Key type lgssc registered [ 5817.293927] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5823.648891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5824.966849] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5825.053953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5828.792494] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 5830.115024] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5835.233714] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5836.627217] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5836.856057] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5840.620684] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 5847.006777] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5852.105716] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5856.961311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5857.244194] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5860.003232] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 5865.127099] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5876.730244] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 5897.188857] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5897.192127] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5897.242224] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5897.247785] Lustre: Skipped 1 previous similar message [ 5901.868249] Lustre: server umount lustre-OST0000 complete [ 5907.430099] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5907.435212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5907.449130] Lustre: Skipped 2 previous similar messages [ 5907.475685] Lustre: Skipped 1 previous similar message [ 5912.545361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5918.689757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5918.694592] Lustre: Skipped 4 previous similar messages [ 5919.200151] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 5919.321508] Lustre: server umount lustre-MDT0000 complete [ 5922.784648] LustreError: 194791:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761317214 with bad export cookie 6204959287253077471 [ 5922.796498] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5922.811407] LustreError: 194791:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5923.168764] Lustre: server umount lustre-MDT0001 complete [ 5932.056403] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 5934.654922] Key type lgssc unregistered [ 5934.880525] LNet: 198491:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5934.887713] LNet: Removed LNI 192.168.202.122@tcp [ 5935.535782] Key type .llcrypt unregistered [ 5935.540923] Key type ._llcrypt unregistered [ 5955.161310] alg: No test for adler32 (adler32-zlib) [ 5955.925618] Key type ._llcrypt registered [ 5955.929711] Key type .llcrypt registered [ 5956.012577] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 5956.789842] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5957.089456] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 5957.240971] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 5957.245693] LNet: Accept secure, port 988 [ 5958.864184] Key type lgssc registered [ 5959.891835] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5966.396278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5967.798844] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5967.838422] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5971.274863] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 5972.964494] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5978.082304] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5978.133353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5978.390265] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5981.648211] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 5987.925982] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5993.262466] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5999.170952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5999.556864] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6002.950794] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 6008.990618] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6018.029024] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 6024.161575] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6024.170403] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6024.183583] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6024.902797] Lustre: lustre-OST0000: Not available for connect from 192.168.202.22@tcp (stopping) [ 6028.259260] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6028.264295] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6028.271253] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6028.274475] Lustre: Skipped 1 previous similar message [ 6030.237849] Lustre: server umount lustre-OST0000 complete [ 6036.170765] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff992345522000.ost_server_uuid 40 [ 6037.766503] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6054.881417] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6054.886168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6054.927873] Lustre: Skipped 1 previous similar message [ 6054.959846] Lustre: Skipped 2 previous similar messages [ 6057.836335] Lustre: server umount lustre-MDT0000 complete [ 6060.004911] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6060.025133] LustreError: Skipped 1 previous similar message [ 6062.064957] LustreError: 199936:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761317354 with bad export cookie 13914983936207717869 [ 6062.070754] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6062.087538] LustreError: 199936:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6062.645923] Lustre: server umount lustre-MDT0001 complete [ 6072.251483] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 10:49:22 (1761317362) [ 6078.792750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6079.162907] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6079.243437] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6083.625564] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 6089.698712] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6089.709484] LustreError: Skipped 1 previous similar message [ 6091.350729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6091.662272] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6095.418577] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 6102.565163] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6107.654485] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6113.071645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6113.340510] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6117.528947] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 6125.164711] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6129.799907] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 6132.952571] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6133.101200] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6133.124619] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 6133.344269] Lustre: lustre-OST0001: new disk, initializing [ 6133.347223] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 6133.447387] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6137.711145] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 6143.054123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6143.065186] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6147.090677] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6150.625295] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6150.644942] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6150.652902] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6156.467160] Lustre: server umount lustre-OST0000 complete [ 6158.003484] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.202.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6161.991976] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 6162.229836] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6166.227094] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 6166.571383] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6179.299194] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6179.300229] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6179.302521] LustreError: Skipped 2 previous similar messages [ 6179.310645] Lustre: Skipped 1 previous similar message [ 6179.317121] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6179.317129] Lustre: Skipped 4 previous similar messages [ 6185.137765] Lustre: server umount lustre-OST0001 complete [ 6189.025538] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6189.029694] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6193.510462] Lustre: server umount lustre-MDT0000 complete [ 6194.658347] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6194.672989] LustreError: Skipped 16 previous similar messages [ 6196.494392] LustreError: 203952:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761317488 with bad export cookie 13914983936207718933 [ 6196.497592] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6196.509302] LustreError: 203952:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6200.786949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6203.461734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6206.249667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6209.150217] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6213.845056] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 10:51:44 (1761317504) [ 6218.951119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6219.123765] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6219.152551] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6219.302677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6222.685643] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 6231.716852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6231.829863] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6231.870221] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6236.621209] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 6244.380843] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6250.202643] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6255.786838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6255.917720] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6259.794108] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 6263.978827] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 6266.528324] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6271.590666] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6271.640165] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6274.970440] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 6281.652089] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6284.264848] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6284.271123] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6284.279107] Lustre: Skipped 1 previous similar message [ 6284.286271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6284.289040] Lustre: Skipped 5 previous similar messages [ 6289.883659] Lustre: server umount lustre-OST0000 complete [ 6289.901896] Lustre: Skipped 1 previous similar message [ 6292.455225] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6292.466947] LustreError: Skipped 1 previous similar message [ 6307.299055] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6307.302742] LustreError: Skipped 1 previous similar message [ 6318.059031] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6318.061701] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6318.073071] Lustre: Skipped 7 previous similar messages [ 6318.078157] Lustre: Skipped 4 previous similar messages [ 6328.288301] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 6328.523164] Lustre: server umount lustre-MDT0000 complete [ 6328.526923] Lustre: Skipped 1 previous similar message [ 6332.725649] LustreError: 210383:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761317624 with bad export cookie 13914983936207720319 [ 6332.733900] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6332.756193] LustreError: 210383:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6338.571722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6341.960131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6346.065597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6349.823408] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6356.328421] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 10:54:06 (1761317646) [ 6371.535516] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_hostid [ 6379.750930] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 6388.182021] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6394.208399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6399.698217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6405.147327] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6410.979470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6411.042064] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6411.220995] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6411.244042] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 6411.307925] Lustre: lustre-MDT0000: new disk, initializing [ 6411.386449] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6411.396905] Lustre: Skipped 3 previous similar messages [ 6411.423451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6415.265946] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 6424.886082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6424.995920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6425.112847] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6425.172724] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 6425.175943] Lustre: Skipped 1 previous similar message [ 6425.234741] Lustre: lustre-MDT0001: new disk, initializing [ 6425.409773] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6425.427965] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6429.806330] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 6439.509450] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6444.613837] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6449.899733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6449.976701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6449.985066] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 6450.193993] Lustre: lustre-OST0000: new disk, initializing [ 6450.204931] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 6453.831358] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 6459.971636] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6459.977387] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6462.974051] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6467.175772] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 6467.444957] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6471.206363] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 6471.470476] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6475.746624] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6475.746760] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6475.769741] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6475.776776] LustreError: Skipped 2 previous similar messages [ 6475.781665] Lustre: Skipped 5 previous similar messages [ 6479.305939] Lustre: server umount lustre-OST0000 complete [ 6479.308037] Lustre: Skipped 1 previous similar message [ 6491.111431] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6491.123695] LustreError: Skipped 11 previous similar messages [ 6491.934904] LustreError: 218624:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761317784 with bad export cookie 13914983936207721831 [ 6491.943756] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6491.951101] LustreError: 218624:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6500.393959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6504.754651] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 6512.850823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6517.550716] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 6524.310560] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6528.911977] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6534.911401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6538.884361] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 6545.399709] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6548.541701] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 6551.864470] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 6555.352603] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 6555.535169] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6558.689688] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6564.052826] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 6568.928265] Lustre: 199270:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761317853/real 1761317853] req@000000000924806e x1846874995795712/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761317860 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 6573.018069] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 6573.025928] Lustre: 199267:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761317858/real 1761317858] req@00000000dbca8986 x1846874995796352/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761317865 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 6573.049585] Lustre: 199267:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 6600.546164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6600.804201] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6600.807534] Lustre: Skipped 5 previous similar messages [ 6602.857776] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6602.872409] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to (at 0@lo) [ 6602.874657] Lustre: lustre-OST0000: deleting orphan objects from 0x0:34 to 0x0:65 [ 6603.775600] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 6608.993663] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6609.002366] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to (at 0@lo) [ 6609.008403] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:65 [ 6609.498971] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6610.772793] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6613.473372] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6613.482983] Lustre: Skipped 5 previous similar messages [ 6613.493135] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6613.495936] Lustre: Skipped 3 previous similar messages [ 6615.530878] Lustre: server umount lustre-OST0000 complete [ 6615.534645] Lustre: Skipped 3 previous similar messages [ 6627.528284] LustreError: 222966:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761317919 with bad export cookie 13914983936207722685 [ 6627.530668] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6627.533223] LustreError: 222966:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6633.720546] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 10:58:44 (1761317924) [ 6637.586790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6640.342987] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 6645.380484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6647.750281] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 6651.827179] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6655.688386] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6659.985501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6663.025355] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 6669.101557] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6672.655338] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 6672.840622] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6675.513268] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 6675.643610] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6679.567100] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6679.609943] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6679.615235] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 6679.687304] Lustre: lustre-OST0001: new disk, initializing [ 6679.690144] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 6682.221597] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 6685.741378] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6685.746878] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6688.467339] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6691.307689] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 6691.426926] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6694.531930] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 6694.681050] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6710.240218] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 6714.512724] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 6714.667901] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6717.734610] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 6717.884456] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6745.223873] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6748.508672] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 6750.699255] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6750.708131] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6750.711426] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to (at 0@lo) [ 6754.308679] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6755.509941] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6760.930143] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6760.935283] LustreError: Skipped 2 previous similar messages [ 6764.203127] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 192.168.202.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6764.208849] LustreError: Skipped 27 previous similar messages [ 6801.378648] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 6804.573835] LustreError: 229214:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761318096 with bad export cookie 13914983936207724099 [ 6804.583795] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6814.696724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6816.954130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6819.630631] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6822.465677] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6827.824467] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 11:01:58 (1761318118) [ 6833.101527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6833.416845] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6833.447903] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6837.559586] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 6843.847862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6843.927365] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6843.942540] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6846.712648] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 6851.705312] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6855.893676] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6860.052964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6860.160164] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6860.298439] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6860.303920] Lustre: Skipped 7 previous similar messages [ 6863.247464] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 6868.550804] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6875.406588] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6875.451293] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6875.537463] Lustre: lustre-OST0001: Not available for connect from 0@lo (not set up) [ 6875.545761] Lustre: Skipped 19 previous similar messages [ 6878.684171] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 6884.527503] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6886.037033] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6889.220187] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 6889.369894] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6892.542594] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 6892.680466] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6896.725823] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff992348d20800.ost_server_uuid 40 [ 6897.790500] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff992348d20800.ost_server_uuid in FULL state after 0 sec [ 6898.451957] Lustre: Permanently deactivating lustre-OST0001 [ 6898.468344] Lustre: Setting parameter lustre-OST0001-osc.osc.active in log lustre-client [ 6900.609781] Lustre: Permanently reactivating lustre-OST0001 [ 6903.787497] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6903.807223] Lustre: Skipped 11 previous similar messages [ 6908.759300] Lustre: server umount lustre-OST0001 complete [ 6908.764917] Lustre: Skipped 7 previous similar messages [ 6926.304150] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 6937.849881] LustreError: 240128:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761318229 with bad export cookie 13914983936207725359 [ 6937.853232] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6937.858602] LustreError: 240128:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 6941.953682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6944.924881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6948.254165] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6952.307658] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6958.826284] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 11:04:09 (1761318249) [ 6959.612421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6965.718386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6966.023876] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6966.044953] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6966.048785] Lustre: Skipped 5 previous similar messages [ 6970.074442] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 6975.262613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6975.348561] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6978.253790] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 6983.046651] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6986.453489] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6990.985382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6991.050583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6991.229694] Lustre: Permanently deactivating lustre-OST0000 [ 6994.793115] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 7000.005339] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7000.086032] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7004.883961] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7007.810194] Lustre: lustre-OST0001: deleting orphan objects from 0x0:67 to 0x0:97 [ 7008.552571] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 7014.467209] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7015.700639] Lustre: Permanently reactivating lustre-OST0000 [ 7015.783740] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7015.790693] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7015.798995] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to (at 0@lo) [ 7015.806093] Lustre: Skipped 1 previous similar message [ 7015.815474] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 7015.828386] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 7028.196286] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7028.201968] LustreError: Skipped 6 previous similar messages [ 7053.585845] LustreError: 244995:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761318345 with bad export cookie 13914983936207726955 [ 7053.589813] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7053.600120] LustreError: 244995:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7061.939623] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 7063.845173] Key type lgssc unregistered [ 7064.080136] LNet: 249955:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7064.089858] LNet: Removed LNI 192.168.202.122@tcp [ 7064.622755] Key type .llcrypt unregistered [ 7064.624413] Key type ._llcrypt unregistered [ 7079.489871] alg: No test for adler32 (adler32-zlib) [ 7080.242908] Key type ._llcrypt registered [ 7080.245162] Key type .llcrypt registered [ 7080.325899] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 7081.030650] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7081.311811] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 7081.463130] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 7081.466190] LNet: Accept secure, port 988 [ 7083.104135] Key type lgssc registered [ 7083.916706] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7086.444803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7091.320596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7092.617937] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7092.673289] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7095.938949] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 7097.832149] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7101.800871] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7101.857205] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7101.916666] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7101.948179] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 7101.952331] Lustre: Permanently deactivating lustre-MDT0001 [ 7102.167340] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7105.077995] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 7110.193517] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7110.619492] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7114.694621] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7120.228878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7120.593902] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7122.662629] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7122.681034] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:193 [ 7124.574368] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 7128.038669] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7130.627685] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7132.134679] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7132.151174] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:163 to 0x280000400:193 [ 7135.008510] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7135.161892] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7138.956264] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 7140.334747] Lustre: lustre-OST0001: deleting orphan objects from 0x0:67 to 0x0:129 [ 7140.338612] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:99 to 0x2c0000400:129 [ 7144.210297] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7146.581367] Lustre: Permanently reactivating lustre-MDT0001 [ 7146.600302] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active in log lustre-client [ 7146.605097] Lustre: Skipped 3 previous similar messages [ 7151.822146] Lustre: lustre-MDT0001: Client 7c30ec4a-a5dd-4550-9b72-3e014af32e94 (at 192.168.202.22@tcp) reconnecting [ 7153.211613] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7153.218264] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7153.226675] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7153.235128] LustreError: 255100:0:(llog_osd.c:2141:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7153.235613] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to (at 0@lo) [ 7153.241660] LustreError: 255100:0:(lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7153.252845] LustreError: 255100:0:(obd_config.c:1999:class_config_llog_handler()) MGC192.168.202.122@tcp: cfg command failed: rc = -108 [ 7153.257770] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7153.257770] [ 7153.261657] LustreError: 251476:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7156.156944] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7157.262289] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7160.698400] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 40 [ 7160.826979] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7164.225620] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 40 [ 7164.418967] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7166.506146] Lustre: Permanently deactivating lustre-MDT0001 [ 7166.517415] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active in log lustre-client [ 7166.520408] Lustre: Skipped 2 previous similar messages [ 7172.651730] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7186.404746] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7186.405920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7186.414432] Lustre: Skipped 3 previous similar messages [ 7186.421835] Lustre: Skipped 2 previous similar messages [ 7191.522456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7191.526734] Lustre: Skipped 2 previous similar messages [ 7191.682929] Lustre: server umount lustre-MDT0000 complete [ 7194.002769] LustreError: 251464:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761318486 with bad export cookie 921186402051791488 [ 7194.005608] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7194.008898] LustreError: 251464:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7194.241698] Lustre: server umount lustre-MDT0001 complete [ 7202.563806] Lustre: server umount lustre-OST0000 complete [ 7210.580171] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 11:08:21 (1761318501) [ 7217.469476] Lustre: server umount lustre-OST0001 complete [ 7221.751978] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_hostid [ 7226.666420] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 7232.765587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7237.318811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7241.784102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7245.630492] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7250.014335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7250.067708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7250.206878] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 7250.212758] Lustre: Skipped 2 previous similar messages [ 7250.232474] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 7250.288515] Lustre: lustre-MDT0000: new disk, initializing [ 7250.349175] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7250.369169] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7253.041291] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 7260.449156] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7260.512638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7260.636928] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 7260.685963] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 7260.696306] Lustre: Skipped 1 previous similar message [ 7260.751577] Lustre: lustre-MDT0001: new disk, initializing [ 7260.811205] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7260.844461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7260.853030] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7264.038806] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 7271.839465] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7275.758622] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7280.354476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7280.444844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7280.454553] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 7280.616279] Lustre: lustre-OST0000: new disk, initializing [ 7280.620097] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 7280.702044] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7284.046715] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 7289.905688] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7289.914126] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7290.704573] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7291.910409] LustreError: 259992:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7297.767794] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7297.808595] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7297.814280] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 7297.891387] Lustre: lustre-OST0001: new disk, initializing [ 7297.896381] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 7300.227940] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 7301.912218] LustreError: 259992:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 awake [ 7301.935321] LustreError: 259992:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7305.291657] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7305.300961] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7307.164282] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7308.493455] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7311.992293] LustreError: 259992:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 awake [ 7315.135622] Lustre: lustre-OST0001: Not available for connect from 192.168.202.22@tcp (stopping) [ 7315.945171] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7315.950924] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7315.959241] Lustre: Skipped 1 previous similar message [ 7315.963180] LustreError: Skipped 1 previous similar message [ 7319.642339] Lustre: server umount lustre-OST0001 complete [ 7320.233535] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 192.168.202.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7320.247939] LustreError: Skipped 1 previous similar message [ 7335.905514] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7335.913124] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7335.923529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7335.926395] Lustre: Skipped 2 previous similar messages [ 7336.426371] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7336.433562] LustreError: Skipped 7 previous similar messages [ 7340.354347] Lustre: server umount lustre-OST0000 complete [ 7346.658055] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7346.660092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7346.667520] Lustre: Skipped 2 previous similar messages [ 7346.673827] Lustre: Skipped 3 previous similar messages [ 7352.406564] LustreError: 263140:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761318644 with bad export cookie 921186402051794267 [ 7352.414601] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7352.418676] LustreError: 263140:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7358.827127] Lustre: server umount lustre-MDT0001 complete [ 7358.830141] Lustre: Skipped 1 previous similar message [ 7366.493304] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 7368.161841] Key type lgssc unregistered [ 7368.359311] LNet: 265180:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7368.365267] LNet: Removed LNI 192.168.202.122@tcp [ 7368.835315] Key type .llcrypt unregistered [ 7368.836820] Key type ._llcrypt unregistered [ 7390.985951] alg: No test for adler32 (adler32-zlib) [ 7391.739430] Key type ._llcrypt registered [ 7391.741320] Key type .llcrypt registered [ 7391.807557] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_hostid [ 7400.593246] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 7401.181053] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7401.354427] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 7401.480179] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 7401.483725] LNet: Accept secure, port 988 [ 7403.099214] Key type lgssc registered [ 7403.729561] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7409.711239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7414.452843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7418.243910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7421.783842] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7425.409708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7425.442844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7426.566025] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 7426.582687] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 7426.627587] Lustre: lustre-MDT0000: new disk, initializing [ 7426.674780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7426.686148] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7428.725639] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 7434.586817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7434.615413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7434.659235] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 7434.686627] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 7434.689151] Lustre: Skipped 1 previous similar message [ 7434.729800] Lustre: lustre-MDT0001: new disk, initializing [ 7434.772936] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7434.788978] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7434.793598] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7436.984333] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 7443.216068] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7447.051480] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7450.902544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7450.952513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7450.957370] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 7451.116791] Lustre: lustre-OST0000: new disk, initializing [ 7451.120056] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 7451.186063] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7453.714886] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 7456.249498] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7456.254309] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7460.131239] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7462.222393] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 7462.311063] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7464.140994] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 7464.241802] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7466.467217] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7466.467275] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7466.467938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7466.474230] Lustre: Skipped 1 previous similar message [ 7466.482243] LustreError: Skipped 1 previous similar message [ 7471.307786] Lustre: server umount lustre-OST0000 complete [ 7475.681194] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7475.685700] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7475.695899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7475.698738] Lustre: Skipped 1 previous similar message [ 7476.705671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7476.706248] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7479.907484] Lustre: server umount lustre-MDT0000 complete [ 7481.825269] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7481.830781] LustreError: Skipped 1 previous similar message [ 7482.423976] LustreError: 270128:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761318774 with bad export cookie 7332800868938324440 [ 7482.425242] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7482.429478] LustreError: 270128:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7482.654184] Lustre: server umount lustre-MDT0001 complete [ 7487.904063] Lustre: DEBUG MARKER: == conf-sanity test 52: check recovering objects from lost+found ========================================================== 11:12:58 (1761318778) [ 7492.260039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7492.577925] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7492.637385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7495.688994] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 7497.702636] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7500.806812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7501.039025] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7503.419656] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 7507.398649] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7510.581734] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7514.355052] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7514.515899] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7516.520828] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 7520.379633] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7533.027553] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7533.029220] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7533.029911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7533.029916] Lustre: Skipped 1 previous similar message [ 7533.033024] Lustre: Skipped 1 previous similar message [ 7534.539047] Lustre: server umount lustre-OST0000 complete [ 7536.681972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7542.752919] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7542.935763] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7545.536592] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 7549.102541] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7549.108229] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to (at 0@lo) [ 7549.108382] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7549.123085] Lustre: lustre-OST0000: deleting orphan objects from 0x0:42 to 0x0:65 [ 7549.127425] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:65 [ 7550.199717] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7559.649867] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7559.650707] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7559.652193] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7559.652199] Lustre: Skipped 1 previous similar message [ 7559.655465] LustreError: Skipped 1 previous similar message [ 7559.663293] Lustre: Skipped 1 previous similar message [ 7561.677663] Lustre: server umount lustre-OST0000 complete [ 7569.783029] Lustre: server umount lustre-MDT0000 complete [ 7569.890020] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7569.899377] LustreError: Skipped 1 previous similar message [ 7571.959679] LustreError: 273809:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761318864 with bad export cookie 7332800868938325294 [ 7571.960130] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7571.967439] LustreError: 273809:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7579.196244] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 7581.024636] Key type lgssc unregistered [ 7581.231019] LNet: 279238:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7581.236941] LNet: Removed LNI 192.168.202.122@tcp [ 7581.720544] Key type .llcrypt unregistered [ 7581.722164] Key type ._llcrypt unregistered [ 7594.969602] alg: No test for adler32 (adler32-zlib) [ 7595.722467] Key type ._llcrypt registered [ 7595.724764] Key type .llcrypt registered [ 7595.782449] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 7596.361369] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7596.546705] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 7596.676409] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 7596.679409] LNet: Accept secure, port 988 [ 7598.304160] Key type lgssc registered [ 7598.904696] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7603.692995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7604.972090] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7605.015663] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7607.521955] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 7610.338417] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7612.185150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7612.386785] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7614.892678] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 7619.918894] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7623.332664] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7626.899185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7627.056802] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7629.221162] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 7633.318074] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7635.244080] Lustre: lustre-OST0000: deleting orphan objects from 0x0:42 to 0x0:97 [ 7653.861261] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7653.864577] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7653.871304] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7655.907554] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7655.909347] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7655.909378] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7655.913548] Lustre: Skipped 1 previous similar message [ 7657.566820] Lustre: server umount lustre-OST0000 complete [ 7661.027722] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7661.034649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7666.145656] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7666.152026] Lustre: Skipped 2 previous similar messages [ 7666.308547] Lustre: server umount lustre-MDT0000 complete [ 7668.525632] LustreError: 280680:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761318960 with bad export cookie 7004488579681653250 [ 7668.529675] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7668.532681] LustreError: 280680:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7668.710727] Lustre: server umount lustre-MDT0001 complete [ 7675.313110] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 7676.655514] Key type lgssc unregistered [ 7676.802547] LNet: 285123:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7676.808707] LNet: Removed LNI 192.168.202.122@tcp [ 7677.257851] Key type .llcrypt unregistered [ 7677.259348] Key type ._llcrypt unregistered [ 7685.661995] alg: No test for adler32 (adler32-zlib) [ 7686.417595] Key type ._llcrypt registered [ 7686.419028] Key type .llcrypt registered [ 7686.471074] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 7686.968883] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7687.152429] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 7687.266673] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 7687.269572] LNet: Accept secure, port 988 [ 7688.872525] Key type lgssc registered [ 7689.390847] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7693.415303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7694.636106] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7694.670843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7696.884464] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 7699.937570] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7701.694752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7701.929768] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7704.088269] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 7708.146062] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7711.030788] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7714.712785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7714.918240] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7717.237518] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 7722.090834] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7725.164751] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 7733.730682] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 7733.735899] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7733.745529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7735.779441] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7735.784322] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7738.965725] Lustre: server umount lustre-OST0000 complete [ 7742.945380] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7742.949255] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7742.955158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7742.957881] Lustre: Skipped 1 previous similar message [ 7746.018248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7746.018529] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7747.416420] Lustre: server umount lustre-MDT0000 complete [ 7749.392939] LustreError: 286371:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761319041 with bad export cookie 14562707594024232473 [ 7749.395402] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7749.402243] LustreError: 286371:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7749.604697] Lustre: server umount lustre-MDT0001 complete [ 7756.194779] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 7757.858536] Key type lgssc unregistered [ 7758.023524] LNet: 290210:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7758.031260] LNet: Removed LNI 192.168.202.122@tcp [ 7758.483915] Key type .llcrypt unregistered [ 7758.485690] Key type ._llcrypt unregistered [ 7768.072976] alg: No test for adler32 (adler32-zlib) [ 7768.824425] Key type ._llcrypt registered [ 7768.826325] Key type .llcrypt registered [ 7768.872798] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 7769.269075] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7769.514810] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 7769.625735] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 7769.633486] LNet: Accept secure, port 988 [ 7771.233187] Key type lgssc registered [ 7771.769755] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7775.396574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7776.666826] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7776.707954] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7779.468465] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 7781.858142] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7784.286782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7784.498026] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7786.904339] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 7791.401762] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7794.901431] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7798.605387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7798.780205] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7801.194990] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 7804.845060] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7806.952723] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 7816.162807] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7816.166295] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7816.172702] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7817.186780] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7817.186987] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7817.191703] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7819.318497] Lustre: server umount lustre-OST0000 complete [ 7822.305904] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7822.309393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7822.311099] Lustre: Skipped 1 previous similar message [ 7822.314348] Lustre: Skipped 1 previous similar message [ 7827.252989] Lustre: server umount lustre-MDT0000 complete [ 7827.429184] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7827.439948] LustreError: Skipped 1 previous similar message [ 7829.258805] LustreError: 291456:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761319121 with bad export cookie 16150774572790869857 [ 7829.261159] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7829.264417] LustreError: 291456:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7829.449574] Lustre: server umount lustre-MDT0001 complete [ 7835.339283] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 7836.828067] Key type lgssc unregistered [ 7836.969913] LNet: 295130:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7836.977590] LNet: Removed LNI 192.168.202.122@tcp [ 7837.374507] Key type .llcrypt unregistered [ 7837.376356] Key type ._llcrypt unregistered [ 7850.894059] alg: No test for adler32 (adler32-zlib) [ 7851.646459] Key type ._llcrypt registered [ 7851.648064] Key type .llcrypt registered [ 7851.698433] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 7852.262798] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7852.431346] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 7852.538207] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 7852.540584] LNet: Accept secure, port 988 [ 7854.144158] Key type lgssc registered [ 7854.618330] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7858.315223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7859.543342] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7859.582953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7861.987622] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 7864.805209] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7866.400186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7866.578358] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7868.503304] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 7872.191733] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7875.226151] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7878.520234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7878.668259] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7880.667553] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 7884.170098] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7889.959902] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:193 [ 7907.298697] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7907.302440] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7907.308091] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7910.374469] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7910.377195] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7910.383485] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7910.393855] Lustre: Skipped 1 previous similar message [ 7911.416211] Lustre: server umount lustre-OST0000 complete [ 7915.490172] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7915.495448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7915.498959] Lustre: Skipped 1 previous similar message [ 7919.252144] Lustre: server umount lustre-MDT0000 complete [ 7920.609084] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7920.614906] LustreError: Skipped 1 previous similar message [ 7921.135934] LustreError: 296571:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761319213 with bad export cookie 16789835491504279133 [ 7921.138446] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7921.141240] LustreError: 296571:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7921.314443] Lustre: server umount lustre-MDT0001 complete [ 7927.034888] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 7928.276411] Key type lgssc unregistered [ 7928.406563] LNet: 301085:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7928.410528] LNet: Removed LNI 192.168.202.122@tcp [ 7928.752525] Key type .llcrypt unregistered [ 7928.754175] Key type ._llcrypt unregistered [ 7937.113777] alg: No test for adler32 (adler32-zlib) [ 7937.866538] Key type ._llcrypt registered [ 7937.868240] Key type .llcrypt registered [ 7937.915707] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 7938.376478] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7938.582658] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 7938.683609] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 7938.685902] LNet: Accept secure, port 988 [ 7940.280364] Key type lgssc registered [ 7940.734983] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7944.070985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7945.320843] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7945.354450] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7947.474196] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 7950.818415] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7951.737722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7951.933660] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7954.075755] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 7957.714850] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7960.339177] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7963.525283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7963.681430] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7965.593639] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 7969.050535] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7974.951717] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:225 [ 7985.121741] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7985.125951] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7985.133790] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7988.194922] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7988.196170] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7988.196202] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7990.560881] Lustre: server umount lustre-OST0000 complete [ 7992.802314] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7992.806101] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7992.812333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7992.815056] Lustre: Skipped 1 previous similar message [ 7998.433381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7998.436747] Lustre: Skipped 3 previous similar messages [ 7998.741206] Lustre: server umount lustre-MDT0000 complete [ 8000.674831] LustreError: 302332:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761319292 with bad export cookie 15452834554690824266 [ 8000.676457] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8000.681260] LustreError: 302332:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8000.867850] Lustre: server umount lustre-MDT0001 complete [ 8007.715713] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 8009.254221] Key type lgssc unregistered [ 8009.398183] LNet: 306208:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8009.402402] LNet: Removed LNI 192.168.202.122@tcp [ 8009.845254] Key type .llcrypt unregistered [ 8009.847057] Key type ._llcrypt unregistered [ 8019.215385] alg: No test for adler32 (adler32-zlib) [ 8019.967431] Key type ._llcrypt registered [ 8019.970288] Key type .llcrypt registered [ 8020.058847] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 8020.669310] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8020.838403] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 8020.944554] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 8020.947186] LNet: Accept secure, port 988 [ 8022.560201] Key type lgssc registered [ 8023.146282] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8027.207428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8028.414970] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8028.445528] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8030.527351] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 8033.761785] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8034.898264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8035.112506] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8037.490296] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 8042.049223] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8045.005894] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8048.156462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8048.314123] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8050.298061] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 8054.098498] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8058.543855] Lustre: lustre-OST0000: deleting orphan objects from 0x0:227 to 0x0:257 [ 8063.971803] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8063.975694] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8063.983210] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8064.641073] Lustre: server umount lustre-OST0000 complete [ 8069.090896] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8069.093214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8069.096485] Lustre: Skipped 2 previous similar messages [ 8069.098684] Lustre: Skipped 1 previous similar message [ 8072.525518] Lustre: server umount lustre-MDT0000 complete [ 8074.152086] LustreError: 309422:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761319366 with bad export cookie 11612552136307447538 [ 8074.155249] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8074.156732] LustreError: 309422:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8074.305255] Lustre: server umount lustre-MDT0001 complete [ 8080.005462] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 8081.140061] Key type lgssc unregistered [ 8081.281365] LNet: 311124:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8081.285694] LNet: Removed LNI 192.168.202.122@tcp [ 8081.620237] Key type .llcrypt unregistered [ 8081.621597] Key type ._llcrypt unregistered [ 8087.506303] alg: No test for adler32 (adler32-zlib) [ 8088.258520] Key type ._llcrypt registered [ 8088.261124] Key type .llcrypt registered [ 8088.322391] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8095.624950] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_hostid [ 8102.142514] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 8102.541932] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8102.797133] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 8102.893702] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 8102.895798] LNet: Accept secure, port 988 [ 8104.488182] Key type lgssc registered [ 8104.901730] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8108.843479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8111.820334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8114.451322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8117.108113] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8119.883080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8119.916368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8121.022781] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 8121.038831] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 8121.080335] Lustre: lustre-MDT0000: new disk, initializing [ 8121.118125] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8121.128178] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8123.014672] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 8128.180779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8128.207067] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8128.255429] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 8128.285068] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 8128.286117] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8128.288469] Lustre: Skipped 1 previous similar message [ 8128.334195] Lustre: lustre-MDT0001: new disk, initializing [ 8128.383072] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8128.402226] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8128.407106] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8130.200834] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 8134.796190] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8137.364674] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8140.225666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8140.252047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8140.256122] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 8140.365097] Lustre: lustre-OST0000: new disk, initializing [ 8140.368252] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 8140.401189] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8142.121633] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 8144.130197] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8144.134947] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8146.720782] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8148.647726] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 8148.746549] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8150.672718] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 8150.775684] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8154.593468] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8154.594255] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8154.597979] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8154.605204] Lustre: Skipped 1 previous similar message [ 8154.608796] Lustre: Skipped 1 previous similar message [ 8157.816990] Lustre: server umount lustre-OST0000 complete [ 8164.321349] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8164.324772] LustreError: Skipped 1 previous similar message [ 8164.326714] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8164.332662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8165.968802] Lustre: server umount lustre-MDT0000 complete [ 8167.695079] LustreError: 314864:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761319459 with bad export cookie 8846819188859649721 [ 8167.697917] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8167.700014] LustreError: 314864:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8167.859776] Lustre: server umount lustre-MDT0001 complete [ 8171.746528] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 11:24:22 (1761319462) [ 8174.645534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8174.867083] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8174.897638] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8176.675758] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 8180.195962] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8180.398784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8180.587378] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8182.372413] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 8185.510165] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8187.899344] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8190.589939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8190.748699] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8192.516912] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 8195.724450] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8201.187086] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8201.187699] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8201.192417] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8201.199129] Lustre: Skipped 2 previous similar messages [ 8201.202972] Lustre: Skipped 1 previous similar message [ 8206.305573] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8206.309222] Lustre: Skipped 2 previous similar messages [ 8207.050157] Lustre: server umount lustre-OST0000 complete [ 8211.172435] Lustre: server umount lustre-MDT0000 complete [ 8211.425887] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8211.426910] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8211.429329] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8211.443026] Lustre: Skipped 1 previous similar message [ 8213.177929] LustreError: 319397:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761319505 with bad export cookie 8846819188859650575 [ 8213.181333] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8213.183295] LustreError: 319397:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8219.136489] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 8220.315924] Key type lgssc unregistered [ 8220.449386] LNet: 323072:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8220.452605] LNet: Removed LNI 192.168.202.122@tcp [ 8220.783655] Key type .llcrypt unregistered [ 8220.784809] Key type ._llcrypt unregistered [ 8233.046313] alg: No test for adler32 (adler32-zlib) [ 8233.798386] Key type ._llcrypt registered [ 8233.799775] Key type .llcrypt registered [ 8233.852167] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_hostid [ 8240.706213] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 8241.155578] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8241.378363] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 8241.470520] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 8241.473444] LNet: Accept secure, port 988 [ 8243.072067] Key type lgssc registered [ 8243.637350] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8248.123169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8251.109281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8254.047073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8257.020249] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8259.470993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8261.995354] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8265.288668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8265.325509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8266.439051] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 8266.455623] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 8266.498979] Lustre: lustre-MDT0000: new disk, initializing [ 8266.550337] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8266.560742] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8268.511616] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 8273.708864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8273.736193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8273.780163] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 8273.800122] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 8273.802932] Lustre: Skipped 1 previous similar message [ 8273.843370] Lustre: lustre-MDT0001: new disk, initializing [ 8273.884675] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8273.906443] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8273.910815] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8275.683582] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 8280.579714] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8283.238764] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8286.032210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8286.076958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8286.082172] Lustre: lustre-OST2710-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 8286.199765] Lustre: lustre-OST2710: new disk, initializing [ 8286.202833] Lustre: srv-lustre-OST2710: No data found on store. Initialize space: rc = -61 [ 8286.251934] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8288.140549] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 8292.784968] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 8294.409351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8294.413666] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8295.796123] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8295.829284] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8295.833253] Lustre: lustre-OST03e8-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 8295.901566] Lustre: lustre-OST03e8: new disk, initializing [ 8295.904505] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space: rc = -61 [ 8295.942898] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8297.879913] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 8299.974666] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8299.981107] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8302.777408] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 8311.096971] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 40 [ 8311.387434] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8317.708559] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 40 [ 8318.196652] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8325.066044] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 40 [ 8325.455543] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8332.039068] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 40 [ 8332.310253] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8339.077211] Lustre: server umount lustre-MDT0000 complete [ 8340.458440] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8340.465803] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8340.482639] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8340.968086] Lustre: lustre-MDT0000-lwp-OST03e8: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8340.988356] Lustre: Skipped 2 previous similar messages [ 8340.998156] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8341.015537] LustreError: Skipped 2 previous similar messages [ 8342.837501] LustreError: 327007:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761319634 with bad export cookie 16735847588080223310 [ 8342.845078] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8342.867746] LustreError: 327007:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8343.274161] Lustre: server umount lustre-MDT0001 complete [ 8353.184196] Lustre: 324814:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761319638/real 1761319638] req@0000000017f30dfc x1846877390663680/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761319645 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 8353.209399] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8353.761364] Lustre: 332501:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761319639/real 1761319639] req@00000000feff033f x1846877390663936/t0(0) o39->lustre-MDT0001-lwp-OST2710@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761319645 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 8353.883674] Lustre: server umount lustre-OST2710 complete [ 8355.233096] Lustre: 324815:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761319640/real 1761319640] req@00000000fa3924b5 x1846877390664064/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761319647 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 8355.269880] Lustre: 324815:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 8357.256383] Lustre: server umount lustre-OST03e8 complete [ 8372.804620] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_hostid [ 8379.881798] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 8387.464980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8392.398393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8396.546505] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8400.680655] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8405.254956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8405.297547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8405.463681] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 8405.482941] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 8405.542794] Lustre: lustre-MDT0000: new disk, initializing [ 8405.618159] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8405.637928] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8409.059213] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 8417.109659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8417.154233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8417.213624] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 8417.246984] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 8417.250142] Lustre: Skipped 1 previous similar message [ 8417.297852] Lustre: lustre-MDT0001: new disk, initializing [ 8417.353234] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8417.376893] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8417.383393] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8420.378463] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 8427.516523] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8431.756277] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8436.459104] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8436.502635] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8436.507890] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 8436.686705] Lustre: lustre-OST0000: new disk, initializing [ 8436.693160] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 8436.767472] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8439.559752] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 8447.037685] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8447.042696] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8447.182377] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8450.242085] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 8450.376724] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8453.364727] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 8453.487939] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8457.699859] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8457.701487] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8457.703153] LustreError: Skipped 1 previous similar message [ 8457.705739] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8457.727287] Lustre: Skipped 1 previous similar message [ 8460.945178] Lustre: server umount lustre-OST0000 complete [ 8467.938316] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8467.939453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8467.946604] Lustre: Skipped 1 previous similar message [ 8467.946885] Lustre: Skipped 1 previous similar message [ 8469.671607] Lustre: server umount lustre-MDT0000 complete [ 8472.488621] LustreError: 339296:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761319764 with bad export cookie 16735847588080224927 [ 8472.492321] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8472.498718] LustreError: 339296:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8479.554070] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 11:29:30 (1761319770) [ 8480.771468] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8482.243844] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 11:29:32 (1761319772) [ 8485.531855] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 8492.846977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8496.586193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8500.315369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8503.658257] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8504.571223] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8510.961182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8511.254976] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8511.280178] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 8511.451858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8515.007966] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 8523.047154] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8523.107698] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8523.127648] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 8526.973551] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 8532.978354] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8537.740227] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8543.653605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8543.761389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8544.107289] Lustre: Denying initial registration attempt from nid 192.168.202.122@tcp, specified as failover [ 8544.127895] LustreError: 160-7: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8544.140589] LustreError: 343633:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -99 [ 8544.149736] LustreError: 343633:0:(obd_mount_server.c:1644:server_put_super()) no obd lustre-OST0000 [ 8544.161616] LustreError: 343633:0:(obd_mount_server.c:131:server_deregister_mount()) lustre-OST0000 not registered [ 8544.345896] Lustre: server umount lustre-OST0000 complete [ 8544.350330] Lustre: Skipped 1 previous similar message [ 8544.353779] LustreError: 343633:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8550.882707] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8550.883268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8550.897214] Lustre: Skipped 1 previous similar message [ 8550.903959] Lustre: Skipped 2 previous similar messages [ 8556.001687] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8556.008247] LustreError: Skipped 1 previous similar message [ 8558.505663] LustreError: 341675:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761319850 with bad export cookie 16735847588080225781 [ 8558.510503] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8569.648387] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 8572.519872] Key type lgssc unregistered [ 8572.747579] LNet: 344707:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8572.755331] LNet: Removed LNI 192.168.202.122@tcp [ 8573.414461] Key type .llcrypt unregistered [ 8573.416222] Key type ._llcrypt unregistered [ 8585.155502] alg: No test for adler32 (adler32-zlib) [ 8585.936035] Key type ._llcrypt registered [ 8585.937177] Key type .llcrypt registered [ 8586.077520] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 8587.002156] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8587.424421] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 8587.592252] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 8587.595755] LNet: Accept secure, port 988 [ 8589.216135] Key type lgssc registered [ 8590.263754] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8597.622152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8601.261494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8604.476704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8607.458504] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8608.249442] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8619.958687] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 8626.122658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8627.437623] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8627.465943] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 8627.595628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8631.164677] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 8638.499941] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8638.560182] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8638.584687] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 8638.806035] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8642.425721] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 8649.479345] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8654.809497] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8660.445386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8660.515629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8660.662110] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8660.828473] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8665.077506] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 8672.038461] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8676.834836] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8676.836253] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8676.836892] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8676.840713] Lustre: Skipped 1 previous similar message [ 8676.856394] Lustre: Skipped 1 previous similar message [ 8680.134146] Lustre: server umount lustre-OST0000 complete [ 8685.026725] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8685.047188] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8685.058880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8687.077670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8687.091965] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8689.597137] Lustre: server umount lustre-MDT0000 complete [ 8692.193306] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8692.204403] LustreError: Skipped 1 previous similar message [ 8693.125348] LustreError: 347584:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761319985 with bad export cookie 10011153376108482368 [ 8693.131168] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8693.511051] Lustre: server umount lustre-MDT0001 complete [ 8702.644553] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 8705.019124] Key type lgssc unregistered [ 8705.222412] LNet: 351287:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8705.227989] LNet: Removed LNI 192.168.202.122@tcp [ 8705.866536] Key type .llcrypt unregistered [ 8705.868197] Key type ._llcrypt unregistered [ 8723.464614] alg: No test for adler32 (adler32-zlib) [ 8724.221405] Key type ._llcrypt registered [ 8724.223057] Key type .llcrypt registered [ 8724.358627] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 8725.284805] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8725.617490] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 8725.779332] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 8725.782096] LNet: Accept secure, port 988 [ 8727.400558] Key type lgssc registered [ 8728.243726] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8733.207823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8734.496595] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8734.549752] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8737.626649] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 8739.812870] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8743.344606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8743.575254] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8746.105735] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 8750.310639] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8754.273837] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8758.409624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8758.621613] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8760.733300] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 8765.072320] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8770.020887] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8770.025105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8770.030032] Lustre: Skipped 2 previous similar messages [ 8771.773321] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.22@tcp (stopping) [ 8771.777217] Lustre: Skipped 2 previous similar messages [ 8772.216409] Lustre: server umount lustre-MDT0000 complete [ 8774.455280] LustreError: 352731:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761320066 with bad export cookie 2344551704939824170 [ 8774.458079] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8774.460532] LustreError: 352731:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8775.137839] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8775.138593] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8775.146554] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8779.169789] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8779.169856] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8779.173939] Lustre: Skipped 1 previous similar message [ 8779.178397] LustreError: Skipped 1 previous similar message [ 8780.754554] Lustre: server umount lustre-MDT0001 complete [ 8785.294642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8790.247617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8791.522811] Lustre: Evicted from MGS (at 192.168.202.122@tcp) after server handle changed from 0x2089855a8a79842a to 0x2089855a8a79a007 [ 8791.527984] Lustre: MGC192.168.202.122@tcp: Connection restored to (at 0@lo) [ 8791.687257] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8791.703441] LustreError: Skipped 1 previous similar message [ 8791.752680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8791.788702] Lustre: lustre-OST0000: deleting orphan objects from 0x0:166 to 0x0:193 [ 8794.208165] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 8799.367290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8799.566341] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8802.157985] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 8806.887397] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 8806.919766] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8807.938067] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8809.959303] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 8811.361946] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8812.270182] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8815.075465] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8815.078283] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8815.090308] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8815.095856] Lustre: Skipped 2 previous similar messages [ 8820.885184] Lustre: server umount lustre-OST0000 complete [ 8825.315367] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8825.315485] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8825.322889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8825.331742] Lustre: Skipped 2 previous similar messages [ 8825.335748] Lustre: Skipped 2 previous similar messages [ 8829.650771] Lustre: server umount lustre-MDT0000 complete [ 8830.435552] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8830.447930] LustreError: Skipped 5 previous similar messages [ 8832.629674] LustreError: 352731:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761320124 with bad export cookie 2344551704939831303 [ 8832.637257] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8832.641783] LustreError: 352731:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8841.789304] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 8843.859691] Key type lgssc unregistered [ 8844.094838] LNet: 359240:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8844.105525] LNet: Removed LNI 192.168.202.122@tcp [ 8844.698956] Key type .llcrypt unregistered [ 8844.700872] Key type ._llcrypt unregistered [ 8862.118108] alg: No test for adler32 (adler32-zlib) [ 8862.871648] Key type ._llcrypt registered [ 8862.877770] Key type .llcrypt registered [ 8863.004045] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 8863.632502] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8864.136293] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 8864.319452] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 8864.325643] LNet: Accept secure, port 988 [ 8865.993045] Key type lgssc registered [ 8866.882370] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8872.852981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8874.191383] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8874.251911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8877.474310] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 8879.590487] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8884.352948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8884.720292] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8888.216254] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 8894.097281] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8897.893149] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8900.065074] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8900.069973] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8900.084376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8904.162361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8904.164373] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8909.291716] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8909.294537] Lustre: Skipped 2 previous similar messages [ 8914.401185] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 8914.401786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8914.417695] Lustre: Skipped 1 previous similar message [ 8914.486138] Lustre: server umount lustre-MDT0000 complete [ 8917.227287] LustreError: 362424:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761320209 with bad export cookie 13247981866719539391 [ 8917.234451] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8917.235766] LustreError: 362424:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8917.457959] Lustre: server umount lustre-MDT0001 complete [ 8926.730795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8926.919113] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8926.944872] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 8927.055293] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8930.131529] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 8936.474373] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8936.566985] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8936.591552] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 8936.767054] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8939.940869] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 8945.064255] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8949.424708] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8954.534129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8954.642193] LustreError: 13b-9: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 8954.648636] LustreError: 160-7: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 8954.655613] LustreError: 365387:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -2 [ 8954.659294] LustreError: 365387:0:(obd_mount_server.c:1644:server_put_super()) no obd lustre-OST0000 [ 8954.662818] LustreError: 365387:0:(obd_mount_server.c:131:server_deregister_mount()) lustre-OST0000 not registered [ 8954.741792] Lustre: server umount lustre-OST0000 complete [ 8954.745214] LustreError: 365387:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 8959.432075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8959.553170] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8959.686091] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8962.702609] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 8968.100492] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8969.292051] Lustre: lustre-OST0000: deleting orphan objects from 0x0:166 to 0x0:225 [ 8973.567125] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8973.626816] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8973.629864] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 8973.665123] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 8973.721384] Lustre: lustre-OST0001: new disk, initializing [ 8973.729884] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 8973.789615] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8975.540195] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8975.550106] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8976.553146] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 8980.676717] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 8986.084995] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8986.086934] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8986.087093] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8986.094730] Lustre: Skipped 1 previous similar message [ 8988.303872] Lustre: server umount lustre-OST0001 complete [ 8991.206458] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8991.207022] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8991.218826] LustreError: Skipped 1 previous similar message [ 8991.224207] Lustre: Skipped 1 previous similar message [ 8996.325504] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8996.328323] Lustre: Skipped 4 previous similar messages [ 8996.329057] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8996.336721] LustreError: Skipped 1 previous similar message [ 8996.620313] Lustre: server umount lustre-OST0000 complete [ 9001.441955] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9001.449250] Lustre: Skipped 1 previous similar message [ 9004.773699] Lustre: server umount lustre-MDT0000 complete [ 9006.562900] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9006.570389] LustreError: Skipped 1 previous similar message [ 9006.838748] LustreError: 363387:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761320298 with bad export cookie 13247981866719539944 [ 9006.839985] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9006.843946] LustreError: 363387:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9012.156896] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 9013.404346] Key type lgssc unregistered [ 9013.540532] LNet: 368753:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9013.544520] LNet: Removed LNI 192.168.202.122@tcp [ 9013.915407] Key type .llcrypt unregistered [ 9013.917153] Key type ._llcrypt unregistered [ 9031.001496] alg: No test for adler32 (adler32-zlib) [ 9031.754429] Key type ._llcrypt registered [ 9031.756063] Key type .llcrypt registered [ 9031.810981] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_hostid [ 9039.780865] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 9040.340461] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9040.529872] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 9040.664833] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 9040.669426] LNet: Accept secure, port 988 [ 9042.288184] Key type lgssc registered [ 9042.841189] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9047.659605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9052.009306] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9055.693249] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9059.194707] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9063.051429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9063.092138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9064.241368] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 9064.261261] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 9064.322878] Lustre: lustre-MDT0000: new disk, initializing [ 9064.390955] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9064.405816] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9066.934943] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 9074.282932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9074.326197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9074.376784] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 9074.401951] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 9074.404837] Lustre: Skipped 1 previous similar message [ 9074.455253] Lustre: lustre-MDT0001: new disk, initializing [ 9074.521363] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9074.564464] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9074.572826] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9077.202636] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 9083.205346] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9086.304527] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9090.420650] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9090.473322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9090.477433] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 9090.650433] Lustre: lustre-OST0000: new disk, initializing [ 9090.653649] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 9090.704209] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9093.261923] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 9096.725233] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9096.729375] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9098.539570] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9100.876450] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 9100.996358] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9103.421915] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 9103.531183] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9106.914307] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9106.916803] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9106.918033] LustreError: Skipped 1 previous similar message [ 9106.918486] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9106.923963] Lustre: Skipped 1 previous similar message [ 9112.036629] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9112.039182] Lustre: Skipped 2 previous similar messages [ 9117.155834] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9117.158580] Lustre: Skipped 1 previous similar message [ 9119.200167] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 9119.324373] Lustre: server umount lustre-OST0000 complete [ 9122.275133] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9122.288698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9127.396114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9127.400922] Lustre: Skipped 2 previous similar messages [ 9127.451066] Lustre: server umount lustre-MDT0000 complete [ 9129.724805] LustreError: 374929:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761320421 with bad export cookie 10596287894404471599 [ 9129.729314] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9129.731771] LustreError: 374929:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9129.988628] Lustre: server umount lustre-MDT0001 complete [ 9134.623682] Lustre: DEBUG MARKER: == conf-sanity test 60: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 11:40:25 (1761320425) [ 9137.806598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9141.226984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9155.863963] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_hostid [ 9161.384139] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 9167.716982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9171.278024] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9174.689905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9178.236186] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9182.412735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9182.465549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9182.640110] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 9182.664860] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 9182.718139] Lustre: lustre-MDT0000: new disk, initializing [ 9182.808181] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9182.825982] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9185.640845] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 9193.178067] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9193.244793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9193.322094] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 9193.365127] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 9193.368721] Lustre: Skipped 1 previous similar message [ 9193.455972] Lustre: lustre-MDT0001: new disk, initializing [ 9193.552115] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9193.582305] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9193.588103] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9196.743709] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 9203.172984] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9207.081231] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9211.690084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9211.755620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9211.759352] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 9211.899728] Lustre: lustre-OST0000: new disk, initializing [ 9211.901712] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 9211.962188] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9214.935780] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 9220.134248] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9220.138265] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9221.613966] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9224.746451] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 9224.947874] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9227.822401] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 9227.986136] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9230.308376] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9230.314371] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9230.328936] Lustre: Skipped 1 previous similar message [ 9230.336405] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9235.370289] Lustre: server umount lustre-OST0000 complete [ 9239.521704] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9239.526583] LustreError: Skipped 1 previous similar message [ 9239.541674] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9239.550019] Lustre: Skipped 1 previous similar message [ 9247.202382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9247.204794] Lustre: Skipped 7 previous similar messages [ 9252.320351] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 9252.391373] Lustre: server umount lustre-MDT0000 complete [ 9254.141866] LustreError: 380604:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761320546 with bad export cookie 10596287894404472453 [ 9254.142841] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9254.146754] LustreError: 380604:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9258.367902] Lustre: DEBUG MARKER: == conf-sanity test 61: large xattr ====================== 11:42:29 (1761320549) [ 9262.791516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9263.076854] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9263.127106] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9265.626452] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 9268.193601] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9270.875547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9273.164126] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 9277.179035] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9280.047175] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9283.510488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9285.776538] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 9289.637617] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9299.007695] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61.conf-sanity [ 9299.883192] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61.conf-sanity [ 9300.732224] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61.conf-sanity [ 9301.577853] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61.conf-sanity after remounting MDS [ 9302.541367] Lustre: Failing over lustre-MDT0000 [ 9302.689968] Lustre: server umount lustre-MDT0000 complete [ 9302.692418] Lustre: Skipped 1 previous similar message [ 9304.547181] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9304.548563] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9304.559846] Lustre: Skipped 3 previous similar messages [ 9304.570283] LustreError: Skipped 2 previous similar messages [ 9309.665403] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9309.674349] LustreError: Skipped 3 previous similar messages [ 9311.715480] Lustre: 371070:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761320596/real 1761320596] req@000000003ce9de19 x1846878228492800/t0(0) o400->MGC192.168.202.122@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761320603 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 9311.725651] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9316.833662] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9316.839722] LustreError: Skipped 7 previous similar messages [ 9317.812731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9328.101978] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9328.104453] Lustre: Evicted from MGS (at 192.168.202.122@tcp) after server handle changed from 0x930d93b2cbc021db to 0x930d93b2cbc025e0 [ 9328.108047] LustreError: Skipped 11 previous similar messages [ 9328.117730] LustreError: 388580:0:(ldlm_resource.c:1127:ldlm_resource_complain()) MGC192.168.202.122@tcp: namespace resource [0x65727473756c:0x5:0x0].0x0 (00000000de828ba5) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 9328.124416] Lustre: MGC192.168.202.122@tcp: Connection restored to 192.168.202.122@tcp (at 0@lo) [ 9328.265334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9328.268776] Lustre: Skipped 2 previous similar messages [ 9328.308488] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9328.808452] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9330.802548] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 9333.733226] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 9333.746726] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 9333.767401] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 9336.658876] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9337.465137] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9339.779846] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61.conf-sanity [ 9340.685864] Lustre: Failing over lustre-MDT0000 [ 9340.791016] Lustre: server umount lustre-MDT0000 complete [ 9343.969758] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9343.970767] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9343.975960] Lustre: Skipped 2 previous similar messages [ 9344.179984] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9344.186093] LustreError: Skipped 3 previous similar messages [ 9351.136174] Lustre: 371067:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761320636/real 1761320636] req@00000000ea6198a9 x1846878228501632/t0(0) o400->MGC192.168.202.122@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761320643 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 9351.149313] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9355.731330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9357.286438] Lustre: Evicted from MGS (at 192.168.202.122@tcp) after server handle changed from 0x930d93b2cbc025e0 to 0x930d93b2cbc028d4 [ 9357.299731] Lustre: MGC192.168.202.122@tcp: Connection restored to (at 0@lo) [ 9357.302213] Lustre: Skipped 2 previous similar messages [ 9357.510844] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9360.308829] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 9362.603462] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9362.919583] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 9362.992365] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 9363.030870] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:97 [ 9367.551810] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9368.732058] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9373.153991] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9373.154797] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9373.155220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9373.155225] Lustre: Skipped 2 previous similar messages [ 9373.159675] Lustre: Skipped 2 previous similar messages [ 9378.331800] Lustre: server umount lustre-MDT0000 complete [ 9380.214172] LustreError: 385241:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761320672 with bad export cookie 10596287894404475092 [ 9380.216705] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9380.220989] LustreError: 385241:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9388.512904] Lustre: 391042:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761320674/real 1761320674] req@00000000e62993a6 x1846878228511360/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761320680 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 9396.877336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9397.150516] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9397.160063] LustreError: Skipped 15 previous similar messages [ 9397.199924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9397.205564] Lustre: Skipped 1 previous similar message [ 9399.275548] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 9403.667429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9406.203777] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 9410.357399] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9413.284966] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9417.025884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9419.527775] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 9423.677599] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9427.439145] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:129 [ 9432.546647] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9432.551071] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9439.713779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9439.717062] Lustre: Skipped 7 previous similar messages [ 9453.024271] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 9453.081795] Lustre: server umount lustre-MDT0000 complete [ 9453.083097] Lustre: Skipped 3 previous similar messages [ 9454.890437] LustreError: 391849:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761320746 with bad export cookie 10596287894404475827 [ 9454.891064] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9454.899922] LustreError: 391849:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9460.751762] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 9462.138309] Key type lgssc unregistered [ 9462.278684] LNet: 395522:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9462.282990] LNet: Removed LNI 192.168.202.122@tcp [ 9462.672626] Key type .llcrypt unregistered [ 9462.674453] Key type ._llcrypt unregistered [ 9474.768306] alg: No test for adler32 (adler32-zlib) [ 9475.520552] Key type ._llcrypt registered [ 9475.523160] Key type .llcrypt registered [ 9475.582594] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 9476.103330] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9476.292764] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 9476.403931] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 9476.406660] LNet: Accept secure, port 988 [ 9478.009140] Key type lgssc registered [ 9478.522399] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9482.280546] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9482.284749] LustreError: 397007:0:(osd_handler.c:8265:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 9482.291107] LustreError: 397007:0:(obd_config.c:774:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 9482.294681] LustreError: 397007:0:(obd_mount.c:200:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 9482.297495] LustreError: 397007:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 9482.301661] LustreError: 397007:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 9485.803714] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9485.807648] LustreError: 397403:0:(osd_handler.c:8265:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 9485.817879] LustreError: 397403:0:(obd_config.c:774:class_setup()) setup lustre-OST0000-osd failed (-22) [ 9485.821294] LustreError: 397403:0:(obd_mount.c:200:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 9485.825060] LustreError: 397403:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 9485.829604] LustreError: 397403:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 9493.661497] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 9495.230554] Key type lgssc unregistered [ 9495.382652] LNet: 398360:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9495.386949] LNet: Removed LNI 192.168.202.122@tcp [ 9495.819572] Key type .llcrypt unregistered [ 9495.821327] Key type ._llcrypt unregistered [ 9505.317478] alg: No test for adler32 (adler32-zlib) [ 9506.069469] Key type ._llcrypt registered [ 9506.071133] Key type .llcrypt registered [ 9506.127087] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_hostid [ 9513.831104] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 9514.268495] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9514.510297] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 9514.622735] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 9514.626083] LNet: Accept secure, port 988 [ 9516.256773] Key type lgssc registered [ 9516.766670] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9521.178458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9524.507859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9527.741585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9530.773366] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9534.199736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9534.227731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9535.350961] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 9535.369671] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 9535.412576] Lustre: lustre-MDT0000: new disk, initializing [ 9535.466405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9535.478712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9537.646304] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 9543.036955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9543.071914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9543.117786] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 9543.142407] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 9543.144536] Lustre: Skipped 1 previous similar message [ 9543.187067] Lustre: lustre-MDT0001: new disk, initializing [ 9543.241085] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9543.257732] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9543.265322] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9545.500718] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 9550.700512] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9553.432568] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9556.600779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9556.662843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9556.667444] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 9556.793777] Lustre: lustre-OST0000: new disk, initializing [ 9556.796925] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 9556.845038] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9558.850054] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 9563.748802] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9565.803714] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 9566.213718] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9566.217775] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9566.957489] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 9569.050120] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 9569.161472] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9571.301679] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9571.302379] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9571.310421] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9571.312265] LustreError: Skipped 1 previous similar message [ 9572.164649] Lustre: server umount lustre-OST0000 complete [ 9576.417978] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9576.419032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9576.423137] Lustre: Skipped 2 previous similar messages [ 9576.425878] Lustre: Skipped 1 previous similar message [ 9581.538026] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9581.541018] Lustre: Skipped 1 previous similar message [ 9586.657556] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9586.660445] Lustre: Skipped 1 previous similar message [ 9588.192251] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 9588.244390] Lustre: server umount lustre-MDT0000 complete [ 9590.255348] LustreError: 401626:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761320882 with bad export cookie 5380864703326516497 [ 9590.257823] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9590.259610] LustreError: 401626:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9590.425054] Lustre: server umount lustre-MDT0001 complete [ 9594.775598] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 9595.631447] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 11:48:06 (1761320886) [ 9598.920671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9599.137500] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9599.176464] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9601.334666] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 9604.577566] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9605.208944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9605.382951] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9607.353140] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 9610.901069] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9613.892666] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9617.294674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9617.451421] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9619.393429] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 9622.851420] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9626.032238] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9626.073212] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9626.077196] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 9626.144230] Lustre: lustre-OST0001: new disk, initializing [ 9626.147127] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 9627.354915] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9627.360410] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9628.178870] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 9633.085620] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 9637.857104] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9637.860400] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9637.861552] LustreError: Skipped 1 previous similar message [ 9637.867213] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9637.874946] Lustre: Skipped 3 previous similar messages [ 9641.344441] Lustre: server umount lustre-OST0001 complete [ 9645.025668] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9645.031468] LustreError: Skipped 1 previous similar message [ 9650.147785] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9650.154589] LustreError: Skipped 1 previous similar message [ 9655.265237] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9655.266378] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9655.267369] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9655.272754] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9655.275125] Lustre: Skipped 2 previous similar messages [ 9655.287026] Lustre: Skipped 4 previous similar messages [ 9660.212440] Lustre: server umount lustre-OST0000 complete [ 9665.508250] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9670.036415] LustreError: 409135:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761320962 with bad export cookie 5380864703326517337 [ 9670.041387] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9670.046202] LustreError: 409135:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9676.296546] Lustre: server umount lustre-MDT0001 complete [ 9676.298561] Lustre: Skipped 1 previous similar message [ 9682.629434] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 9683.980754] Key type lgssc unregistered [ 9684.122876] LNet: 411135:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9684.126711] LNet: Removed LNI 192.168.202.122@tcp [ 9684.522945] Key type .llcrypt unregistered [ 9684.524360] Key type ._llcrypt unregistered [ 9699.378509] alg: No test for adler32 (adler32-zlib) [ 9700.130411] Key type ._llcrypt registered [ 9700.131398] Key type .llcrypt registered [ 9700.184231] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_hostid [ 9707.434579] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 9707.906662] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9708.081162] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 9708.173641] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 9708.176191] LNet: Accept secure, port 988 [ 9709.768103] Key type lgssc registered [ 9710.255120] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9714.478647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9717.722720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9720.512597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9723.432847] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9726.635278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9726.668174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9727.805923] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 9727.834749] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 9727.883471] Lustre: lustre-MDT0000: new disk, initializing [ 9727.937486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9727.950315] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9730.028538] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 9735.797098] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9735.842334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9735.887460] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 9735.912914] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 9735.916119] Lustre: Skipped 1 previous similar message [ 9735.958422] Lustre: lustre-MDT0001: new disk, initializing [ 9736.007432] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9736.026306] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9736.032046] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9738.037427] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 9743.017890] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9745.695975] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9749.029755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9749.072287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9749.079453] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 9749.200579] Lustre: lustre-OST0000: new disk, initializing [ 9749.203380] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 9749.242577] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9751.233145] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 9756.052881] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9756.675308] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9756.679468] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9758.066034] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 9758.179206] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9760.203559] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 9760.306455] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9761.762888] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9761.768099] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9761.775301] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9761.778457] Lustre: Skipped 1 previous similar message [ 9763.317859] Lustre: server umount lustre-OST0000 complete [ 9766.881895] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9766.882173] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9766.883302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9766.887038] Lustre: Skipped 2 previous similar messages [ 9766.890374] LustreError: Skipped 1 previous similar message [ 9771.143404] Lustre: server umount lustre-MDT0000 complete [ 9772.001224] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9772.006775] LustreError: Skipped 1 previous similar message [ 9772.814307] LustreError: 417311:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761321064 with bad export cookie 13547862199097002497 [ 9772.817137] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9772.819263] LustreError: 417311:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9772.976372] Lustre: server umount lustre-MDT0001 complete [ 9776.720511] Lustre: DEBUG MARKER: == conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================================================== 11:51:07 (1761321067) [ 9780.843497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9784.711647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9784.916280] Lustre: lustre-MDT0000: new disk, initializing [ 9784.930846] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9784.963686] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9786.948606] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 9790.433943] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9790.812668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9790.984689] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9792.997325] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 9796.348964] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9798.822409] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9801.184976] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9801.186155] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9801.189294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9801.192476] Lustre: Skipped 1 previous similar message [ 9801.196480] Lustre: Skipped 2 previous similar messages [ 9806.305454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9806.308200] Lustre: Skipped 1 previous similar message [ 9811.425311] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9811.427940] Lustre: Skipped 1 previous similar message [ 9813.984174] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 9814.029945] Lustre: server umount lustre-MDT0000 complete [ 9815.625278] LustreError: 420352:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761321107 with bad export cookie 13547862199097003337 [ 9815.629358] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9815.630321] LustreError: 420352:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9819.573431] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 11:51:50 (1761321110) [ 9822.434552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9822.639089] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9822.672398] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9824.591367] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 9827.815363] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9828.677574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9831.055929] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 9834.472355] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9837.006641] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9840.081730] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9840.240966] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9840.245096] Lustre: Skipped 1 previous similar message [ 9842.127150] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 9845.384147] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9859.520941] Lustre: Permanently deactivating lustre-OST0000 [ 9859.529708] Lustre: Setting parameter lustre-OST0000-osc.osc.active in log lustre-client [ 9869.298329] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 9870.127450] LustreError: 426181:0:(mgs_llog.c:1551:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002020 [ 9870.132607] LustreError: 426181:0:(mgs_llog.c:1605:mgs_replace_nids()) Only MGS is allowed to be started [ 9870.135820] LustreError: 426181:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -115 [ 9871.102299] LustreError: 426229:0:(mgs_llog.c:1605:mgs_replace_nids()) Only MGS is allowed to be started [ 9871.105535] LustreError: 426229:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -115 [ 9878.203949] Lustre: server umount lustre-OST0000 complete [ 9878.205792] Lustre: Skipped 1 previous similar message [ 9879.434520] LustreError: 426479:0:(mgs_llog.c:1605:mgs_replace_nids()) Only MGS is allowed to be started [ 9879.438236] LustreError: 426479:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -115 [ 9883.619548] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9883.620479] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9883.625641] Lustre: Skipped 1 previous similar message [ 9886.688172] Lustre: 426577:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761321172/real 1761321172] req@00000000b2da434e x1846878928933824/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761321178 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 9886.775809] Lustre: server umount lustre-MDT0000 complete [ 9888.403418] LustreError: 422947:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761321180 with bad export cookie 13547862199097003876 [ 9888.406346] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9888.407966] LustreError: 422947:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9894.880288] Lustre: 426778:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761321180/real 1761321180] req@000000006f90d11f x1846878928934656/t0(0) o9->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761321186 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 9899.290972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9901.291527] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 9902.588842] LustreError: 427686:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -6 [ 9905.478257] Lustre: 428032:0:(mgs_llog.c:1311:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.122@tcp [ 9912.800207] Lustre: 428131:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761321198/real 1761321198] req@000000004d4b738e x1846878928935040/t0(0) o251->MGC192.168.202.122@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761321204 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 9912.816165] Lustre: server umount MGS complete [ 9912.817800] Lustre: Skipped 1 previous similar message [ 9917.702653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9917.921555] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9917.928668] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 9917.931549] Lustre: Skipped 1 previous similar message [ 9917.953579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9919.803512] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 9923.322572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9923.445646] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 9925.258826] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 9928.484175] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9930.958882] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9931.769469] Lustre: Permanently reactivating lustre-OST0000 [ 9931.776792] Lustre: Modifying parameter lustre-OST0000-osc.osc.active in log lustre-client [ 9931.780223] Lustre: Skipped 2 previous similar messages [ 9947.671226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9949.709842] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [ 9953.019146] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9969.637114] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9969.637748] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9969.638242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9969.638251] Lustre: Skipped 1 previous similar message [ 9969.648079] Lustre: Skipped 1 previous similar message [ 9971.442204] Lustre: server umount lustre-OST0000 complete [ 9974.755059] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9974.759146] LustreError: Skipped 1 previous similar message [ 9979.873374] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9979.879524] LustreError: Skipped 2 previous similar messages [ 9980.753961] LustreError: 428738:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761321272 with bad export cookie 13547862199097004821 [ 9980.758200] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9980.759427] LustreError: 428738:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9986.349536] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [ 9987.618277] Key type lgssc unregistered [ 9987.751539] LNet: 433021:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9987.755303] LNet: Removed LNI 192.168.202.122@tcp [ 9988.105948] Key type .llcrypt unregistered [ 9988.107525] Key type ._llcrypt unregistered [ 9995.736121] alg: No test for adler32 (adler32-zlib) [ 9996.488473] Key type ._llcrypt registered [ 9996.489754] Key type .llcrypt registered [ 9996.536269] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_hostid [10003.586741] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [10004.072890] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10004.236614] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [10004.335033] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [10004.337136] LNet: Accept secure, port 988 [10005.936148] Key type lgssc registered [10006.376615] Lustre: Echo OBD driver; http://www.lustre.org/ [10010.302178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10013.132967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10015.680426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10018.219302] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10020.955145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10020.983091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10022.077773] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [10022.091386] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [10022.133362] Lustre: lustre-MDT0000: new disk, initializing [10022.170245] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10022.179214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10023.836257] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [10028.479069] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10028.505590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10028.550766] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [10028.573679] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [10028.577147] Lustre: Skipped 1 previous similar message [10028.618595] Lustre: lustre-MDT0001: new disk, initializing [10028.663668] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10028.680480] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10028.685272] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10030.590582] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [10035.121591] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10037.348586] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10040.089599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10040.122171] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10040.125615] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [10040.255086] Lustre: lustre-OST0000: new disk, initializing [10040.258217] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [10040.292871] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10042.166398] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [10046.799756] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10048.590206] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [10049.029128] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10049.033491] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10049.713560] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [10051.569423] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [10051.667845] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10054.114878] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10054.115108] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10054.116710] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10054.122135] LustreError: Skipped 1 previous similar message [10054.127270] Lustre: Skipped 1 previous similar message [10058.625935] Lustre: server umount lustre-OST0000 complete [10064.353400] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10064.354472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10064.358936] Lustre: Skipped 1 previous similar message [10064.361808] Lustre: Skipped 1 previous similar message [10066.566931] Lustre: server umount lustre-MDT0000 complete [10068.206512] LustreError: 436285:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761321360 with bad export cookie 11091254113017807732 [10068.209121] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10068.211632] LustreError: 436285:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [10068.371523] Lustre: server umount lustre-MDT0001 complete [10071.891664] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [10072.650528] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 11:56:03 (1761321363) [10075.541186] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10075.753747] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10075.787637] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10077.613020] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [10081.163297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10081.249139] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10081.334955] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10083.212911] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [10086.426399] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10088.945066] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10091.824703] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10091.975070] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10093.852669] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [10096.782674] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10097.540464] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [10108.691714] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [10111.457266] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10111.458327] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10111.460541] LustreError: Skipped 1 previous similar message [10111.465655] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10111.469219] Lustre: Skipped 1 previous similar message [10116.582899] Lustre: server umount lustre-OST0000 complete [10121.698139] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10121.698831] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10121.703303] Lustre: Skipped 2 previous similar messages [10121.707634] Lustre: Skipped 1 previous similar message [10124.235858] Lustre: server umount lustre-MDT0000 complete [10125.769117] LustreError: 440880:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761321417 with bad export cookie 11091254113017808572 [10125.771096] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10125.773483] LustreError: 440880:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [10131.340116] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [10132.673971] Key type lgssc unregistered [10132.807634] LNet: 444763:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10132.812142] LNet: Removed LNI 192.168.202.122@tcp [10133.178488] Key type .llcrypt unregistered [10133.180237] Key type ._llcrypt unregistered [10143.417406] alg: No test for adler32 (adler32-zlib) [10144.170454] Key type ._llcrypt registered [10144.172147] Key type .llcrypt registered [10144.224195] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [10144.343404] Key type .llcrypt unregistered [10144.344951] Key type ._llcrypt unregistered [10151.907219] alg: No test for adler32 (adler32-zlib) [10152.659464] Key type ._llcrypt registered [10152.660866] Key type .llcrypt registered [10152.703604] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [10153.147405] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10153.305663] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [10153.396150] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [10153.398341] LNet: Accept secure, port 988 [10154.984162] Key type lgssc registered [10155.385338] Lustre: Echo OBD driver; http://www.lustre.org/ [10158.383659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10159.541935] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10159.562059] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10161.254304] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [10164.556849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10164.683573] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10164.688759] LustreError: Skipped 1 previous similar message [10164.691219] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10166.349141] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [10166.771547] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [10169.317488] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10169.826228] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10169.831522] LustreError: Skipped 1 previous similar message [10172.138747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10172.309982] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10174.149465] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [10182.112853] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10182.116301] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10182.121763] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10182.629494] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10182.629618] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10182.638061] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10187.746538] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10187.748979] Lustre: Skipped 1 previous similar message [10187.941145] Lustre: server umount lustre-OST0000 complete [10191.843340] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [10191.846755] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10191.852180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10195.550096] Lustre: server umount lustre-MDT0000 complete [10197.058515] LustreError: 447116:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761321489 with bad export cookie 11344329887268470263 [10197.062937] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10197.063522] LustreError: 447116:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [10197.213462] Lustre: server umount lustre-MDT0001 complete [10202.382021] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [10203.554956] Key type lgssc unregistered [10203.686454] LNet: 450353:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10203.690179] LNet: Removed LNI 192.168.202.122@tcp [10204.025511] Key type .llcrypt unregistered [10204.027096] Key type ._llcrypt unregistered [10214.355277] alg: No test for adler32 (adler32-zlib) [10215.107440] Key type ._llcrypt registered [10215.109123] Key type .llcrypt registered [10215.158587] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [10215.656447] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10215.822988] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [10215.927961] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [10215.930897] LNet: Accept secure, port 988 [10217.528179] Key type lgssc registered [10217.951295] Lustre: Echo OBD driver; http://www.lustre.org/ [10221.203815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10238.688465] LustreError: 451787:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.122@tcp: failed processing log, type 1: rc = -5 [10269.408345] LustreError: 451787:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.122@tcp: failed processing log, type 4: rc = -110 [10299.104276] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10300.919679] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [10303.979811] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10306.886941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10321.971293] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10322.006737] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10322.040697] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:97 [10323.920080] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [10327.010090] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10327.482098] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10327.655119] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10329.418761] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [10332.602707] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10335.143297] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10338.276049] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10338.276940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10338.283587] Lustre: Skipped 1 previous similar message [10343.393529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10343.396775] Lustre: Skipped 1 previous similar message [10344.369313] Lustre: server umount lustre-OST0000 complete [10348.001431] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10348.004692] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10348.005458] LustreError: Skipped 1 previous similar message [10348.012839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10348.186513] Lustre: server umount lustre-MDT0000 complete [10348.515512] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10349.918615] LustreError: 451795:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761321641 with bad export cookie 3577889885655879377 [10349.921486] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10349.923314] LustreError: 451795:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [10350.076372] Lustre: server umount lustre-MDT0001 complete [10355.404068] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [10356.495418] Key type lgssc unregistered [10356.618519] LNet: 455477:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10356.622674] LNet: Removed LNI 192.168.202.122@tcp [10356.966890] Key type .llcrypt unregistered [10356.968542] Key type ._llcrypt unregistered [10366.358993] alg: No test for adler32 (adler32-zlib) [10367.111354] Key type ._llcrypt registered [10367.112864] Key type .llcrypt registered [10367.154614] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [10367.613538] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10367.772870] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [10367.857833] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [10367.860066] LNet: Accept secure, port 988 [10369.448118] Key type lgssc registered [10369.830400] Lustre: Echo OBD driver; http://www.lustre.org/ [10372.756806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10373.906719] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10373.924885] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10375.558395] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [10378.750274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10378.902215] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10380.621340] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [10383.993247] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10386.385779] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10389.100930] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10389.233204] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10390.246249] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:129 [10390.911928] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [10393.977435] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10399.713339] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10399.717261] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10399.723207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10400.433611] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.22@tcp (stopping) [10400.738181] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10400.742733] Lustre: Skipped 1 previous similar message [10402.184104] Lustre: server umount lustre-MDT0000 complete [10403.077050] LustreError: 457751:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.22@tcp arrived at 1761321695 with bad export cookie 4258465396377236152 [10403.082188] LustreError: 457751:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [10403.782234] LustreError: 457751:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761321695 with bad export cookie 4258465396377235277 [10403.786157] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10403.787253] LustreError: 457751:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [10405.857300] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10405.857316] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10405.857874] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10405.858610] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10405.858614] Lustre: Skipped 3 previous similar messages [10405.862586] LustreError: Skipped 1 previous similar message [10408.864886] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10408.864968] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10408.870343] LustreError: Skipped 1 previous similar message [10409.893582] Lustre: server umount lustre-OST0000 complete [10412.478377] Lustre: server umount lustre-MDT0001 complete [10417.485282] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [10418.579922] Key type lgssc unregistered [10418.689416] LNet: 460768:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10418.692174] LNet: Removed LNI 192.168.202.122@tcp [10418.982641] Key type .llcrypt unregistered [10418.984037] Key type ._llcrypt unregistered [10428.401852] alg: No test for adler32 (adler32-zlib) [10429.153324] Key type ._llcrypt registered [10429.154340] Key type .llcrypt registered [10429.191250] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [10429.609655] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10429.748574] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [10429.835384] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [10429.837311] LNet: Accept secure, port 988 [10431.424174] Key type lgssc registered [10431.825302] Lustre: Echo OBD driver; http://www.lustre.org/ [10434.580796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10435.729295] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10435.745070] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10437.246373] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [10440.237059] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10440.373018] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10441.843108] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [10444.504216] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10446.438751] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10448.839835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10448.973210] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10450.604058] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [10451.049077] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:161 [10453.333663] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10457.057145] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [10457.061188] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10457.066827] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10459.815700] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.22@tcp (stopping) [10460.644293] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10461.388411] Lustre: server umount lustre-MDT0001 complete [10465.761173] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10465.761714] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10465.771376] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10465.774182] Lustre: Skipped 2 previous similar messages [10469.421751] Lustre: server umount lustre-OST0000 complete [10470.881656] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10477.536154] Lustre: 465279:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761321763/real 1761321763] req@000000006313cc14 x1846879684931200/t0(0) o251->MGC192.168.202.122@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761321769 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [10477.733888] Lustre: server umount lustre-MDT0000 complete [10494.252468] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [10498.470568] Key type lgssc unregistered [10498.937147] LNet: 466036:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10498.947273] LNet: Removed LNI 192.168.202.122@tcp [10500.185828] Key type .llcrypt unregistered [10500.193573] Key type ._llcrypt unregistered [10523.995482] alg: No test for adler32 (adler32-zlib) [10524.750552] Key type ._llcrypt registered [10524.752700] Key type .llcrypt registered [10524.822232] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_hostid [10532.333652] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [10532.939883] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10533.125145] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [10533.223327] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [10533.226048] LNet: Accept secure, port 988 [10534.816128] Key type lgssc registered [10535.332548] Lustre: Echo OBD driver; http://www.lustre.org/ [10539.846286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10543.244844] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10546.377894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10550.089444] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10554.732389] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [10559.076562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10562.691961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10562.731152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10563.847298] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [10563.867392] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [10563.919814] Lustre: lustre-MDT0000: new disk, initializing [10563.986046] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10563.999279] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10566.681517] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [10572.416156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10576.470415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10576.517900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10576.521507] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [10576.650987] Lustre: lustre-OST0000: new disk, initializing [10576.653965] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [10576.657501] Lustre: Skipped 1 previous similar message [10576.694201] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10579.127981] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [10584.587854] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10588.994104] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10592.643389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10592.693168] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10592.753505] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [10592.789115] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [10592.831306] Lustre: lustre-MDT0001: new disk, initializing [10592.872533] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [10592.882894] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [10592.908668] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10595.181022] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [10600.120309] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 40 [10600.232805] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [10602.276319] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 40 [10603.405867] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 1 sec [10608.097854] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [10608.098480] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10608.100142] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10608.111031] Lustre: Skipped 1 previous similar message [10609.335287] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.22@tcp (stopping) [10609.337670] Lustre: Skipped 1 previous similar message [10611.705318] Lustre: server umount lustre-MDT0001 complete [10613.219285] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10613.224714] LustreError: Skipped 1 previous similar message [10618.339266] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10618.343258] LustreError: Skipped 1 previous similar message [10623.461494] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10623.468699] LustreError: Skipped 1 previous similar message [10628.067788] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10628.071311] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10628.076471] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10628.580158] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10633.697940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10633.701210] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10638.821585] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10639.328148] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [10639.451059] Lustre: server umount lustre-OST0000 complete [10656.226524] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [10662.368383] Lustre: 474448:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761321948/real 1761321948] req@000000007d7865fb x1846879793988480/t0(0) o251->MGC192.168.202.122@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761321954 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [10662.497865] Lustre: server umount lustre-MDT0000 complete [10670.711548] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [10672.101714] Key type lgssc unregistered [10672.262628] LNet: 475205:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10672.270670] LNet: Removed LNI 192.168.202.122@tcp [10672.661677] Key type .llcrypt unregistered [10672.663354] Key type ._llcrypt unregistered [10692.785473] alg: No test for adler32 (adler32-zlib) [10693.539521] Key type ._llcrypt registered [10693.541167] Key type .llcrypt registered [10693.598666] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [10693.744435] Key type .llcrypt unregistered [10693.746253] Key type ._llcrypt unregistered [10702.613306] alg: No test for adler32 (adler32-zlib) [10703.365507] Key type ._llcrypt registered [10703.367028] Key type .llcrypt registered [10703.419341] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [10703.816483] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10704.098488] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [10704.190153] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [10704.193119] LNet: Accept secure, port 988 [10705.792293] Key type lgssc registered [10706.335463] Lustre: Echo OBD driver; http://www.lustre.org/ [10710.257073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10713.380756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10716.445513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10720.062772] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10720.102241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10721.217795] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [10721.238751] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [10721.289224] Lustre: lustre-MDT0000: new disk, initializing [10721.353090] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10721.367439] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10723.582761] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [10729.532666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10729.573957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10729.634773] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [10729.666106] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [10729.669258] Lustre: Skipped 1 previous similar message [10729.719861] Lustre: lustre-MDT0001: new disk, initializing [10729.781452] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10729.808666] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10729.813782] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10732.235981] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [10738.096335] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10741.332823] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10744.901613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10744.936355] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10744.940598] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [10745.056554] Lustre: lustre-OST0000: new disk, initializing [10745.059507] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [10745.103034] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10747.191782] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [10747.280330] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10747.285489] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10752.973400] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10757.602858] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10757.605673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10757.610582] Lustre: Skipped 2 previous similar messages [10757.613652] Lustre: Skipped 1 previous similar message [10762.267828] Lustre: server umount lustre-MDT0000 complete [10762.723437] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10762.730672] LustreError: Skipped 2 previous similar messages [10764.186929] LustreError: 478688:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761322056 with bad export cookie 4964733639462636303 [10764.188875] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10764.191710] LustreError: 478688:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [10764.374735] Lustre: server umount lustre-MDT0001 complete [10772.448934] Lustre: 481942:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761322058/real 1761322058] req@00000000221b483e x1846879973292672/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761322064 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [10772.591462] Lustre: server umount lustre-OST0000 complete [10778.297430] Lustre: DEBUG MARKER: == conf-sanity test 73: failnode to update from mountdata properly ========================================================== 12:07:49 (1761322069) [10778.822629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10782.526901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10782.786724] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10782.826228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10785.103693] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [10788.322143] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10789.559429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10789.735110] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10791.930360] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [10795.719799] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10798.558312] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10801.708587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10801.744200] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10801.839444] Lustre: Found index 0 for lustre-OST0000, updating log [10801.926255] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10803.943889] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [10807.852638] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10813.228061] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5 to 0x0:33 [10818.530901] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10818.531460] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10818.536370] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10818.540675] Lustre: Skipped 2 previous similar messages [10821.310956] Lustre: server umount lustre-OST0000 complete [10823.649333] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10823.653713] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10823.655775] Lustre: Skipped 2 previous similar messages [10826.083462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10826.086232] Lustre: Skipped 2 previous similar messages [10831.329027] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10831.331646] Lustre: Skipped 1 previous similar message [10832.233857] Lustre: server umount lustre-MDT0000 complete [10833.977595] LustreError: 482747:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761322126 with bad export cookie 4964733639462637640 [10833.980080] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10833.983111] LustreError: 482747:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [10838.250241] Lustre: DEBUG MARKER: == conf-sanity test 75: The order of --index should be irrelevant ========================================================== 12:08:49 (1761322129) [10841.219096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10843.664887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10846.341153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10848.685690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10856.373533] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_hostid [10860.621949] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [10865.188095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10868.145575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10870.877438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10873.748100] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10877.068042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10877.098291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10877.232873] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [10877.251448] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [10877.294709] Lustre: lustre-MDT0000: new disk, initializing [10877.349099] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10877.359339] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10879.536624] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [10885.382114] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10885.415468] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10885.466632] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [10885.505846] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [10885.512043] Lustre: Skipped 1 previous similar message [10885.514314] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10885.557915] Lustre: lustre-MDT0001: new disk, initializing [10885.621289] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10885.628256] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10887.827769] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [10893.152746] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10895.920832] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10899.151330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10899.206160] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10899.211364] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [10899.365478] Lustre: lustre-OST0000: new disk, initializing [10899.368179] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [10901.254136] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug -1 all 8 [10902.855223] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10902.859119] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10906.130262] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10908.132591] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [10908.233316] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10910.305220] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [10910.414772] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10913.250479] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10913.251450] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10913.254578] LustreError: Skipped 1 previous similar message [10913.255219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10913.263265] Lustre: Skipped 1 previous similar message [10917.358062] Lustre: server umount lustre-OST0000 complete [10917.360506] Lustre: Skipped 1 previous similar message [10921.440893] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10921.443912] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10927.171783] LustreError: 491864:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761322219 with bad export cookie 4964733639462638697 [10927.172331] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10927.178077] LustreError: 491864:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [10938.790680] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_hostid [10943.067456] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [10948.043052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10951.727365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10954.447527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10957.232875] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10957.942648] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 10723 sec ========= 12:10:49 (1761322249) [10974.513991] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [10975.743807] Key type lgssc unregistered [10975.864581] LNet: 498505:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10975.868674] LNet: Removed LNI 192.168.202.122@tcp [10976.204615] Key type .llcrypt unregistered [10976.206076] Key type ._llcrypt unregistered