[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 3.0.0 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-2.fc40 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 490905029 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f53f0-0x000f53ff] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5200 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1D87 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1C23 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001BE3 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1C97 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1D27 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1D5F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1c23-0xbffe1c96] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1c22] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1c97-0xbffe1d26] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1d27-0xbffe1d5e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1d5f-0xbffe1d86] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002350] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004015] kvm-guest: setup PV IPIs [ 0.006942] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008000] pid_max: default: 32768 minimum: 301 [ 0.008000] LSM: Security Framework initializing [ 0.008047] Yama: becoming mindful. [ 0.009049] SELinux: Initializing. [ 0.010090] *** VALIDATE selinux *** [ 0.019237] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024311] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025160] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027044] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028099] *** VALIDATE tmpfs *** [ 0.029430] *** VALIDATE proc *** [ 0.031194] *** VALIDATE cgroup *** [ 0.032008] *** VALIDATE cgroup2 *** [ 0.033235] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034136] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037004] Spectre V2 : User space: Vulnerable [ 0.038007] Speculative Store Bypass: Vulnerable [ 0.041383] debug: unmapping init [mem 0xffffffffa4259000-0xffffffffa4260fff] [ 0.043169] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044624] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045020] ... version: 2 [ 0.046012] ... bit width: 48 [ 0.047009] ... generic registers: 4 [ 0.048008] ... value mask: 0000ffffffffffff [ 0.049010] ... max period: 00007fffffffffff [ 0.050010] ... fixed-purpose events: 3 [ 0.051009] ... event mask: 000000070000000f [ 0.052277] rcu: Hierarchical SRCU implementation. [ 0.054483] smp: Bringing up secondary CPUs ... [ 0.055543] x86: Booting SMP configuration: [ 0.056019] .... node #0, CPUs: #1 #2 #3 [ 0.067112] smp: Brought up 1 node, 4 CPUs [ 0.069024] smpboot: Max logical packages: 1 [ 0.070011] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.287917] node 0 deferred pages initialised in 215ms [ 0.291142] devtmpfs: initialized [ 0.292490] x86/mm: Memory block size: 128MB [ 0.296967] gcov: version magic: 0x41383552 [ 0.299265] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.304248] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.307302] pinctrl core: initialized pinctrl subsystem [ 0.309173] [ 0.309773] ************************************************************* [ 0.312016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.315013] ** ** [ 0.317013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.320014] ** ** [ 0.322012] ** This means that this kernel is built to expose internal ** [ 0.325015] ** IOMMU data structures, which may compromise security on ** [ 0.326010] ** your system. ** [ 0.329011] ** ** [ 0.331012] ** If you see this message and you are not debugging the ** [ 0.333012] ** kernel, report this immediately to your vendor! ** [ 0.335015] ** ** [ 0.338013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.340014] ************************************************************* [ 0.343702] NET: Registered protocol family 16 [ 0.345446] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.348081] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.351069] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.355048] cpuidle: using governor menu [ 0.356689] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.359739] PCI: Using configuration type 1 for base access [ 0.362154] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.371130] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.374077] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.378100] cryptd: max_cpu_qlen set to 1000 [ 0.381238] ACPI: Added _OSI(Module Device) [ 0.382018] ACPI: Added _OSI(Processor Device) [ 0.384026] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.385012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.389628] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.398198] ACPI: Interpreter enabled [ 0.400100] ACPI: PM: (supports S0 S3 S4 S5) [ 0.401019] ACPI: Using IOAPIC for interrupt routing [ 0.403197] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.407492] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.419427] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.422061] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.425092] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.428105] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.434417] acpiphp: Slot [2] registered [ 0.436221] acpiphp: Slot [5] registered [ 0.437186] acpiphp: Slot [6] registered [ 0.439213] acpiphp: Slot [7] registered [ 0.440185] acpiphp: Slot [8] registered [ 0.442148] acpiphp: Slot [9] registered [ 0.444191] acpiphp: Slot [10] registered [ 0.446210] acpiphp: Slot [3] registered [ 0.447134] acpiphp: Slot [4] registered [ 0.449156] acpiphp: Slot [11] registered [ 0.450117] acpiphp: Slot [12] registered [ 0.451132] acpiphp: Slot [13] registered [ 0.452118] acpiphp: Slot [14] registered [ 0.454130] acpiphp: Slot [15] registered [ 0.455121] acpiphp: Slot [16] registered [ 0.457142] acpiphp: Slot [17] registered [ 0.458140] acpiphp: Slot [18] registered [ 0.460131] acpiphp: Slot [19] registered [ 0.461122] acpiphp: Slot [20] registered [ 0.463141] acpiphp: Slot [21] registered [ 0.465178] acpiphp: Slot [22] registered [ 0.466128] acpiphp: Slot [23] registered [ 0.468117] acpiphp: Slot [24] registered [ 0.469094] acpiphp: Slot [25] registered [ 0.470065] acpiphp: Slot [26] registered [ 0.471115] acpiphp: Slot [27] registered [ 0.473103] acpiphp: Slot [28] registered [ 0.474095] acpiphp: Slot [29] registered [ 0.476103] acpiphp: Slot [30] registered [ 0.477134] acpiphp: Slot [31] registered [ 0.479077] PCI host bridge to bus 0000:00 [ 0.481025] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.483026] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.486027] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.489027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.491024] pci_bus 0000:00: root bus resource [mem 0x380000000000-0x38007fffffff window] [ 0.494028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.497219] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.500024] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.503517] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.513026] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.519075] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.522024] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.524024] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.526028] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.528649] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.531805] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.534077] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.537396] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.542015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.556019] pci 0000:00:02.0: reg 0x20: [mem 0x380000000000-0x380000003fff 64bit pref] [ 0.563014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.569844] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.576027] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.582023] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.599027] pci 0000:00:05.0: reg 0x20: [mem 0x380000004000-0x380000007fff 64bit pref] [ 0.608625] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.616019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.622022] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.635036] pci 0000:00:06.0: reg 0x20: [mem 0x380000008000-0x38000000bfff 64bit pref] [ 0.645000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.648016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.652018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.663018] pci 0000:00:07.0: reg 0x20: [mem 0x38000000c000-0x38000000ffff 64bit pref] [ 0.674925] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.680018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.684017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.694019] pci 0000:00:08.0: reg 0x20: [mem 0x380000010000-0x380000013fff 64bit pref] [ 0.705276] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.710019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.716016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.730014] pci 0000:00:09.0: reg 0x20: [mem 0x380000014000-0x380000017fff 64bit pref] [ 0.739462] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.747017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.753018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.767084] pci 0000:00:0a.0: reg 0x20: [mem 0x380000018000-0x38000001bfff 64bit pref] [ 0.778979] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.781384] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.784378] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.786365] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.788230] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.793031] iommu: Default domain type: Passthrough [ 0.795586] SCSI subsystem initialized [ 0.797202] ACPI: bus type USB registered [ 0.798185] usbcore: registered new interface driver usbfs [ 0.801089] usbcore: registered new interface driver hub [ 0.802000] usbcore: registered new device driver usb [ 0.803187] pps_core: LinuxPPS API ver. 1 registered [ 0.806013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.809059] PTP clock support registered [ 0.811153] EDAC MC: Ver: 3.0.0 [ 0.813688] PCI: Using ACPI for IRQ routing [ 0.815799] NetLabel: Initializing [ 0.817013] NetLabel: domain hash size = 128 [ 0.818009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.820084] NetLabel: unlabeled traffic allowed by default [ 0.822159] vgaarb: loaded [ 0.824250] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.826013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.831000] clocksource: Switched to clocksource kvm-clock [ 0.942539] VFS: Disk quotas dquot_6.6.0 [ 0.944223] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.946996] *** VALIDATE ramfs *** [ 0.948456] *** VALIDATE hugetlbfs *** [ 0.951089] pnp: PnP ACPI init [ 0.953569] pnp: PnP ACPI: found 6 devices [ 0.974048] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.977709] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.980134] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.983044] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.985641] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.988287] pci_bus 0000:00: resource 8 [mem 0x380000000000-0x38007fffffff window] [ 0.991294] NET: Registered protocol family 2 [ 0.993554] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.999061] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.002364] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.009031] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.012673] TCP: Hash tables configured (established 65536 bind 65536) [ 1.015242] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.018626] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.021493] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.024412] NET: Registered protocol family 1 [ 1.027585] RPC: Registered named UNIX socket transport module. [ 1.029887] RPC: Registered udp transport module. [ 1.031819] RPC: Registered tcp transport module. [ 1.033723] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.035878] NET: Registered protocol family 44 [ 1.037721] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.040015] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.042033] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.044296] PCI: CLS 0 bytes, default 64 [ 1.046014] Unpacking initramfs... [ 2.488489] debug: unmapping init [mem 0xffff9f837cc54000-0xffff9f837ffbffff] [ 2.492413] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.494736] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.497681] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.037692] Initialise system trusted keyrings [ 3.039655] Key type blacklist registered [ 3.044208] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.053230] zbud: loaded [ 3.056398] *** VALIDATE nfs *** [ 3.057585] *** VALIDATE nfs4 *** [ 3.059329] pstore: using deflate compression [ 3.062586] Platform Keyring initialized [ 3.170286] NET: Registered protocol family 38 [ 3.172063] Key type asymmetric registered [ 3.173785] Asymmetric key parser 'x509' registered [ 3.175867] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.179722] io scheduler mq-deadline registered [ 3.181055] io scheduler kyber registered [ 3.182801] io scheduler bfq registered [ 3.184247] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.186811] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.189853] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.192605] ACPI: Power Button [PWRF] [ 3.284601] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.375279] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.554781] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.645144] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.831473] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.860345] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.889813] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.905896] Non-volatile memory driver v1.3 [ 3.907859] Linux agpgart interface v0.103 [ 3.944652] virtio_blk virtio1: [vda] 67976 512-byte logical blocks (34.8 MB/33.2 MiB) [ 3.948916] vda: detected capacity change from 0 to 34803712 [ 3.965062] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.968085] vdb: detected capacity change from 0 to 1073741824 [ 3.989942] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.992581] vdc: detected capacity change from 0 to 2621440000 [ 4.009098] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.011795] vdd: detected capacity change from 0 to 2621440000 [ 4.030759] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.033431] vde: detected capacity change from 0 to 4294967296 [ 4.048215] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.051336] vdf: detected capacity change from 0 to 4294967296 [ 4.057811] libphy: Fixed MDIO Bus: probed [ 4.063781] usbcore: registered new interface driver usbserial_generic [ 4.066386] usbserial: USB Serial support registered for generic [ 4.068858] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.072750] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.074252] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.076689] mousedev: PS/2 mouse device common for all mice [ 4.079740] rtc_cmos 00:05: RTC can wake from S4 [ 4.085229] rtc_cmos 00:05: registered as rtc0 [ 4.086766] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.089339] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.097318] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.099327] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.100599] intel_pstate: CPU model not supported [ 4.108192] hid: raw HID events driver (C) Jiri Kosina [ 4.110445] usbcore: registered new interface driver usbhid [ 4.112656] usbhid: USB HID core driver [ 4.114818] drop_monitor: Initializing network drop monitor service [ 4.117444] Initializing XFRM netlink socket [ 4.119385] NET: Registered protocol family 10 [ 4.122505] Segment Routing with IPv6 [ 4.124137] NET: Registered protocol family 17 [ 4.126411] mpls_gso: MPLS GSO support [ 4.131212] RAS: Correctable Errors collector initialized. [ 4.133624] AVX version of gcm_enc/dec engaged. [ 4.135536] AES CTR mode by8 optimization enabled [ 4.220176] sched_clock: Marking stable (4220133705, 0)->(5156205610, -936071905) [ 4.223168] registered taskstats version 1 [ 4.224854] Loading compiled-in X.509 certificates [ 4.226606] zswap: loaded using pool lzo/zbud [ 4.257639] Key type big_key registered [ 4.270075] Key type encrypted registered [ 4.271913] ima: No TPM chip found, activating TPM-bypass! [ 4.274371] ima: Allocated hash algorithm: sha1 [ 4.276851] ima: No architecture policies found [ 4.278754] evm: Initialising EVM extended attributes: [ 4.280764] evm: security.selinux [ 4.282080] evm: security.ima [ 4.283249] evm: security.capability [ 4.284337] evm: HMAC attrs: 0x1 [ 4.286596] rtc_cmos 00:05: setting system clock to 2025-10-24 14:26:53 UTC (1761316013) [ 4.292691] debug: unmapping init [mem 0xffffffffa5203000-0xffffffffa53fffff] [ 4.295986] debug: unmapping init [mem 0xffffffffa3f82000-0xffffffffa4258fff] [ 4.306078] Write protecting the kernel read-only data: 28672k [ 4.309433] debug: unmapping init [mem 0xffffffffa2603000-0xffffffffa27fffff] [ 4.312414] debug: unmapping init [mem 0xffffffffa2f14000-0xffffffffa2ffffff] [ 4.355357] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.364634] systemd[1]: Detected virtualization kvm. [ 4.366651] systemd[1]: Detected architecture x86-64. [ 4.368705] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.397283] systemd[1]: No hostname configured. [ 4.398834] systemd[1]: Set hostname to . [ 4.401051] random: systemd: uninitialized urandom read (16 bytes read) [ 4.403743] systemd[1]: Initializing machine ID from random generator. [ 4.458641] random: ln: uninitialized urandom read (6 bytes read) [ 4.554161] random: systemd: uninitialized urandom read (16 bytes read) [ 4.556498] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 4.561344] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 4.565644] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Reached target Slices. Starting Create Volatile Files and Directories... [ OK ] Reached target Swap. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.248108] device-mapper: uevent: version 1.0.3 [ 5.250307] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 6.008166] virtio_net virtio0 ens2: renamed from eth0 [ 6.034590] scsi host0: ata_piix [ 6.034827] random: fast init done [ 6.082230] scsi host1: ata_piix [ 6.083688] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 6.085683] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.564030] dracut-initqueue[590]: RTNETLINK answers: File exists [ 10.789491] random: crng init done [ 10.790813] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 11.352203] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.598375] printk: systemd: 25 output lines suppressed due to ratelimiting [ 12.852131] SELinux: Disabled at runtime. [ 12.912840] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.921324] systemd[1]: Detected virtualization kvm. [ 12.923466] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.388421] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.391899] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.397104] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.403049] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.406661] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.415598] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.424558] systemd[1]: Starting Apply Kernel Variables... Starting Apply Kernel Variables... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target Slices. [ OK ] Reached target rpc_pipefs.target. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Huge Pages File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Starting Create list of required st…ce nodes for the current kernel... Mounting POSIX Message Queue File System... Mounting Kernel Debug File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target Paths. [ 13.582495] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-getty.slice. [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 13.879353] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 14.196918] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.326948] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.412087] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.445801] EDAC sbridge: Ver: 1.1.2 [ 16.223674] Key type dns_resolver registered [ 16.522590] NFS: Registering the id_resolver key type [ 16.524705] Key type id_resolver registered [ 16.526389] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg151-server login: [ 39.204223] libcfs: loading out-of-tree module taints kernel. [ 39.223805] alg: No test for adler32 (adler32-zlib) [ 39.975491] Key type ._llcrypt registered [ 39.977197] Key type .llcrypt registered [ 40.024753] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_hostid [ 48.237439] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 48.851889] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 49.131184] Lustre: Lustre: Build Version: 2.15.7_10_g040485f [ 49.439438] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 49.442318] LNet: Accept secure, port 988 [ 51.071176] Key type lgssc registered [ 51.700909] Lustre: Echo OBD driver; http://www.lustre.org/ [ 63.724461] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 65.707900] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 71.481458] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 77.191752] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 83.431816] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 86.418726] hrtimer: interrupt took 4015260 ns [ 96.352520] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 107.359440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 107.454884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 108.796711] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 108.843404] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 109.037312] Lustre: lustre-MDT0000: new disk, initializing [ 109.202756] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 109.236281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 114.103808] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 125.579921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 125.655617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 125.745194] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 125.788285] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 125.791335] Lustre: Skipped 1 previous similar message [ 125.877572] Lustre: lustre-MDT0001: new disk, initializing [ 125.945277] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 125.985643] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 126.004317] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 129.555507] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 140.868308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 140.913380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 140.916920] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 141.156286] Lustre: lustre-OST0000: new disk, initializing [ 141.161535] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 141.213077] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 143.503900] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 143.507663] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 144.895367] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 156.025540] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 156.088955] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 156.092890] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 156.157755] Lustre: lustre-OST0001: new disk, initializing [ 156.160583] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 156.196825] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 159.789674] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 168.695481] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 173.383310] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 181.979410] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing check_logdir /tmp/testlogs/ [ 185.758139] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing yml_node [ 189.516909] Lustre: DEBUG MARKER: Client: 2.15.7.10 [ 191.731958] Lustre: DEBUG MARKER: MDS: 2.15.7.10 [ 193.714799] Lustre: DEBUG MARKER: OSS: 2.15.7.10 [ 194.975613] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Oct 24 10:30:02 EDT 2025 [ 201.543851] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball 110 [ 202.759494] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 [ 218.081697] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 218.086300] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 218.093064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 221.155907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 221.156696] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 221.165714] Lustre: Skipped 1 previous similar message [ 226.272752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 226.280985] Lustre: Skipped 6 previous similar messages [ 230.371548] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 230.464594] Lustre: server umount lustre-MDT0000 complete [ 231.392830] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 231.410038] LustreError: Skipped 3 previous similar messages [ 232.934993] LustreError: 9194:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761316242 with bad export cookie 11558034628378599850 [ 232.938542] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 232.943066] LustreError: 9194:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 236.512316] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 236.515315] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 236.524328] LustreError: Skipped 1 previous similar message [ 236.529686] Lustre: Skipped 1 previous similar message [ 236.539213] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 236.547795] Lustre: Skipped 1 previous similar message [ 237.535697] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 237.544835] LustreError: Skipped 1 previous similar message [ 239.584412] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 239.592063] LustreError: Skipped 1 previous similar message [ 244.708846] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 244.710371] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 244.722822] Lustre: Skipped 5 previous similar messages [ 247.263803] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 247.458082] Lustre: server umount lustre-MDT0001 complete [ 250.257909] Lustre: server umount lustre-OST0000 complete [ 253.882990] Lustre: server umount lustre-OST0001 complete [ 260.398458] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_hostid [ 266.608774] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 273.249279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 277.855407] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 281.934801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 285.490483] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 288.963761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 289.005809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 289.127669] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 289.145279] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 289.189379] Lustre: lustre-MDT0000: new disk, initializing [ 289.238177] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 289.251415] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 291.384534] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 298.146803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 298.194574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 298.269587] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 298.309403] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 298.313235] Lustre: Skipped 1 previous similar message [ 298.381426] Lustre: lustre-MDT0001: new disk, initializing [ 298.450041] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 298.478766] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 298.486108] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 301.304720] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 307.893581] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 311.640249] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 315.290494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 315.339067] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 315.344891] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 315.515857] Lustre: lustre-OST0000: new disk, initializing [ 315.521268] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 315.590576] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 317.335688] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 317.344159] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 318.060386] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 324.066657] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 326.938037] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 327.075391] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 329.559661] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 329.690966] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 332.771702] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 332.772507] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 332.775372] LustreError: Skipped 1 previous similar message [ 332.786443] Lustre: Skipped 1 previous similar message [ 332.788858] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 332.794030] Lustre: Skipped 1 previous similar message [ 345.056974] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 345.167087] Lustre: server umount lustre-OST0000 complete [ 348.129082] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 348.135252] Lustre: Skipped 2 previous similar messages [ 353.250431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 353.252970] Lustre: Skipped 6 previous similar messages [ 353.519585] Lustre: server umount lustre-MDT0000 complete [ 355.930840] LustreError: 14300:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761316365 with bad export cookie 11558034628378601306 [ 355.935187] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 355.936494] LustreError: 14300:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 358.638736] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 10:32:45 (1761316365) [ 360.305947] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 361.447374] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 10:32:48 (1761316368) [ 365.911044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 366.198868] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 366.219497] LustreError: Skipped 1 previous similar message [ 366.262130] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 369.449822] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 375.106829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 377.983736] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 382.980665] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 387.215220] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 391.529633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 391.650644] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 391.662031] LustreError: Skipped 1 previous similar message [ 394.485447] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 399.731366] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 406.496397] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 406.500648] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 406.506080] Lustre: Skipped 1 previous similar message [ 406.506369] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 410.916348] Lustre: server umount lustre-OST0000 complete [ 410.918604] Lustre: Skipped 1 previous similar message [ 416.225672] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 422.856986] LustreError: 18839:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761316432 with bad export cookie 11558034628378602160 [ 422.869769] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 432.205681] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 434.402878] Key type lgssc unregistered [ 434.616910] LNet: 22510:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 434.631791] LNet: Removed LNI 192.168.201.151@tcp [ 435.339776] Key type .llcrypt unregistered [ 435.341289] Key type ._llcrypt unregistered [ 452.957503] alg: No test for adler32 (adler32-zlib) [ 453.720485] Key type ._llcrypt registered [ 453.722387] Key type .llcrypt registered [ 453.793103] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 454.688904] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 454.950480] Lustre: Lustre: Build Version: 2.15.7_10_g040485f [ 455.077243] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 455.081435] LNet: Accept secure, port 988 [ 456.703114] Key type lgssc registered [ 457.610955] Lustre: Echo OBD driver; http://www.lustre.org/ [ 464.143196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 465.518516] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 465.556527] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 469.378952] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 471.009202] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 476.136437] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 477.921402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 478.248970] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 481.920496] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 488.466503] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 492.638730] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 497.530321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 498.121712] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 501.838903] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 509.617322] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 516.611510] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 528.865190] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 528.868273] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 528.875399] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 531.938038] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 531.939630] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 531.954798] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 533.518157] Lustre: server umount lustre-OST0000 complete [ 537.057091] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 537.059103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 537.064384] Lustre: Skipped 1 previous similar message [ 537.075117] Lustre: Skipped 2 previous similar messages [ 542.179334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 542.184455] Lustre: Skipped 1 previous similar message [ 542.420868] Lustre: server umount lustre-MDT0000 complete [ 545.625446] LustreError: 23953:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761316554 with bad export cookie 3158385086805699278 [ 545.638065] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 545.644157] LustreError: 23953:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 545.854850] Lustre: server umount lustre-MDT0001 complete [ 554.780446] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 557.020513] Key type lgssc unregistered [ 557.205319] LNet: 27649:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 557.213267] LNet: Removed LNI 192.168.201.151@tcp [ 557.808588] Key type .llcrypt unregistered [ 557.809854] Key type ._llcrypt unregistered [ 574.674307] alg: No test for adler32 (adler32-zlib) [ 575.427405] Key type ._llcrypt registered [ 575.430972] Key type .llcrypt registered [ 575.496866] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 575.962477] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 576.262977] Lustre: Lustre: Build Version: 2.15.7_10_g040485f [ 576.384436] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 576.388993] LNet: Accept secure, port 988 [ 578.039180] Key type lgssc registered [ 578.710096] Lustre: Echo OBD driver; http://www.lustre.org/ [ 584.701504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 586.040083] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 586.089100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 589.501787] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 591.330943] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 596.450791] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 596.674883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 597.010382] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 600.398255] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 605.373185] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 608.961716] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 613.244208] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 613.512158] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 616.565117] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 616.651431] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 622.781691] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 632.800445] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 632.807621] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 632.816211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 633.316139] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 633.323134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 633.338732] Lustre: Skipped 1 previous similar message [ 633.340744] Lustre: Skipped 1 previous similar message [ 634.732086] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.51@tcp (stopping) [ 634.735296] Lustre: Skipped 1 previous similar message [ 637.941907] Lustre: server umount lustre-MDT0000 complete [ 638.437572] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 638.444621] LustreError: Skipped 2 previous similar messages [ 640.813902] LustreError: 29090:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761316650 with bad export cookie 16149647111252045876 [ 640.816531] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 640.823110] LustreError: 29090:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 643.553258] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 643.553808] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 643.564364] LustreError: Skipped 1 previous similar message [ 643.572444] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 647.655509] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 647.659115] Lustre: Skipped 3 previous similar messages [ 652.129717] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.201.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 652.143048] LustreError: Skipped 4 previous similar messages [ 655.332456] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 655.545765] Lustre: server umount lustre-MDT0001 complete [ 668.576902] Lustre: server umount lustre-OST0000 complete [ 680.084467] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 682.143794] Key type lgssc unregistered [ 682.427585] LNet: 33092:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 682.437828] LNet: Removed LNI 192.168.201.151@tcp [ 683.057427] Key type .llcrypt unregistered [ 683.058847] Key type ._llcrypt unregistered [ 699.709643] alg: No test for adler32 (adler32-zlib) [ 700.462491] Key type ._llcrypt registered [ 700.464427] Key type .llcrypt registered [ 700.518562] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 701.037751] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 701.248160] Lustre: Lustre: Build Version: 2.15.7_10_g040485f [ 701.392545] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 701.400549] LNet: Accept secure, port 988 [ 703.015119] Key type lgssc registered [ 703.630751] Lustre: Echo OBD driver; http://www.lustre.org/ [ 708.702317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 709.969915] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 710.018727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 712.249605] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 715.238544] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 716.854085] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 717.059394] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 719.495186] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 723.944411] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 727.153177] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 731.432996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 731.672595] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 734.789245] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 739.890292] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 740.625816] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 747.380599] Lustre: lustre-OST0000: Not available for connect from 192.168.201.51@tcp (stopping) [ 748.000673] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 748.021223] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 748.053112] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 750.053764] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 750.056673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 750.061561] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 750.073170] Lustre: Skipped 1 previous similar message [ 752.292331] Lustre: server umount lustre-OST0000 complete [ 768.481218] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 768.485651] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 768.494794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 771.554684] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 774.272921] Lustre: server umount lustre-MDT0000 complete [ 776.673805] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 776.697604] LustreError: Skipped 1 previous similar message [ 777.146819] LustreError: 34535:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761316786 with bad export cookie 2578660330939783266 [ 777.149596] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 777.158408] LustreError: 34535:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 783.436514] Lustre: server umount lustre-MDT0001 complete [ 790.739564] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 792.367931] Key type lgssc unregistered [ 792.553371] LNet: 38381:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 792.559372] LNet: Removed LNI 192.168.201.151@tcp [ 793.072449] Key type .llcrypt unregistered [ 793.074755] Key type ._llcrypt unregistered [ 807.958094] alg: No test for adler32 (adler32-zlib) [ 808.711552] Key type ._llcrypt registered [ 808.714181] Key type .llcrypt registered [ 808.782161] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 809.449692] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 809.676509] Lustre: Lustre: Build Version: 2.15.7_10_g040485f [ 809.856087] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 809.861162] LNet: Accept secure, port 988 [ 811.472661] Key type lgssc registered [ 812.086542] Lustre: Echo OBD driver; http://www.lustre.org/ [ 817.473130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 818.783395] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 818.840487] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 821.854232] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 824.290372] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 827.684919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 827.894315] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 830.811651] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 836.933457] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 842.145556] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 847.784842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 848.024188] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 849.078181] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 851.526474] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 857.411535] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 861.939745] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 865.137729] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 873.970291] LNetError: 120-3: Refusing connection from 192.168.201.151 for 127.0.0.2@tcp: No matching NI [ 873.977605] LNetError: 39145:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 873.980195] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 877.486894] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2025-10-2410h41m23s [ 878.890561] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 884.192216] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 884.193488] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 884.194452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 884.220413] Lustre: Skipped 2 previous similar messages [ 884.223383] LNetError: 120-3: Refusing connection from 192.168.201.151 for 127.0.0.2@tcp: No matching NI [ 884.228529] LNetError: 39142:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 884.243978] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 887.174793] Lustre: server umount lustre-MDT0000 complete [ 889.312721] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 889.328977] LNetError: 120-3: Refusing connection from 192.168.201.151 for 127.0.0.2@tcp: No matching NI [ 889.331293] LustreError: Skipped 2 previous similar messages [ 889.336977] LNetError: 39143:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 889.356144] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 890.184675] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 894.440142] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 894.447631] LNetError: 120-3: Refusing connection from 192.168.201.151 for 127.0.0.2@tcp: No matching NI [ 894.458169] LustreError: Skipped 3 previous similar messages [ 894.464974] LNetError: 39144:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 894.474777] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 895.862105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 895.994091] LustreError: 11-0: MGC192.168.201.151@tcp: operation mgs_target_reg to node 0@lo failed: rc = -107 [ 895.997576] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 896.005118] Lustre: Evicted from MGS (at 192.168.201.151@tcp) after server handle changed from 0x3d959c91784c7b83 to 0x3d959c91784c7fce [ 896.017875] Lustre: MGC192.168.201.151@tcp: Connection restored to 192.168.201.151@tcp (at 0@lo) [ 896.229430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 896.310279] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:193 [ 899.790571] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 901.612448] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 901.624090] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 902.825500] Lustre: DEBUG MARKER: Wait for df (28865) ... [ 904.049620] Lustre: DEBUG MARKER: done [ 909.794332] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 909.798422] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 909.804812] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 909.811743] Lustre: Skipped 3 previous similar messages [ 911.841483] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 911.844211] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 911.848457] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 913.083117] Lustre: server umount lustre-OST0000 complete [ 916.960985] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 916.962644] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 916.968902] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 916.973698] Lustre: Skipped 1 previous similar message [ 916.980518] LNetError: 120-3: Refusing connection from 192.168.201.151 for 127.0.0.2@tcp: No matching NI [ 916.983022] Lustre: Skipped 2 previous similar messages [ 916.988702] LNetError: 39145:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 917.000903] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 922.080941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 922.086784] Lustre: Skipped 1 previous similar message [ 930.276911] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 930.397138] Lustre: server umount lustre-MDT0000 complete [ 932.330648] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 932.346839] LustreError: Skipped 6 previous similar messages [ 933.382621] LustreError: 42541:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761316942 with bad export cookie 4437625156445962190 [ 933.385457] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 933.388418] LustreError: 42541:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 933.645789] Lustre: server umount lustre-MDT0001 complete [ 942.723759] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 945.124438] Key type lgssc unregistered [ 945.404902] LNet: 44943:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 945.414853] LNet: Removed LNI 192.168.201.151@tcp [ 946.019430] Key type .llcrypt unregistered [ 946.024895] Key type ._llcrypt unregistered [ 973.218244] alg: No test for adler32 (adler32-zlib) [ 973.973236] Key type ._llcrypt registered [ 973.978190] Key type .llcrypt registered [ 974.067416] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_hostid [ 985.246563] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 985.887561] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 986.177989] Lustre: Lustre: Build Version: 2.15.7_10_g040485f [ 986.321706] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 986.325404] LNet: Accept secure, port 988 [ 987.951168] Key type lgssc registered [ 988.640692] Lustre: Echo OBD driver; http://www.lustre.org/ [ 995.007380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1000.249691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1005.155788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1010.145735] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1015.340692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1015.401897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1016.644486] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1016.687722] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1016.784123] Lustre: lustre-MDT0000: new disk, initializing [ 1016.858701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1016.878628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1020.579624] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 1030.506983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1030.582344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1030.677931] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1030.716450] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1030.719578] Lustre: Skipped 1 previous similar message [ 1030.784866] Lustre: lustre-MDT0001: new disk, initializing [ 1030.838235] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1030.862440] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1030.868283] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1033.937186] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 1041.841951] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1047.922451] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1054.389960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1054.449065] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1054.461828] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1054.739843] Lustre: lustre-OST0000: new disk, initializing [ 1054.765423] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1054.877979] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1059.514474] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 1061.442566] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1061.454898] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1068.209280] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1071.941785] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1072.115153] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1075.979625] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 1076.176363] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1082.338675] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1082.349541] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1082.368259] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1082.378886] Lustre: Skipped 1 previous similar message [ 1087.458403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1087.461589] Lustre: Skipped 1 previous similar message [ 1092.063101] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1092.187685] Lustre: server umount lustre-OST0000 complete [ 1097.697713] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1097.701953] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1097.714053] Lustre: Skipped 2 previous similar messages [ 1101.754550] Lustre: server umount lustre-MDT0000 complete [ 1102.816553] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1102.827730] LustreError: Skipped 1 previous similar message [ 1105.247963] LustreError: 49036:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761317114 with bad export cookie 5517662009532086395 [ 1105.250922] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1105.261863] LustreError: 49036:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1105.694244] Lustre: server umount lustre-MDT0001 complete [ 1113.884600] Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 10:45:20 (1761317120) [ 1120.001130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1120.288846] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1120.350860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1123.351208] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 1125.346408] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1129.207220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1129.449539] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1132.536205] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 1138.870566] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1143.787628] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1150.844365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1151.178692] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1155.072107] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 1161.374390] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1176.931163] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2025-10-2410h46m23s [ 1178.170249] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1184.616248] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1186.836502] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1186.891681] LustreError: 53589:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1189.356923] LNetError: 120-3: Refusing connection from 192.168.201.151 for 127.0.0.2@tcp: No matching NI [ 1189.375032] LNetError: 47251:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1189.389185] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1207.164405] Lustre: lustre-MDT0000: Client fff5241e-5fa3-4075-bdc5-796aedae5cc0 (at 192.168.201.51@tcp) reconnecting [ 1226.895153] LustreError: 53589:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 136 awake [ 1229.394528] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1232.366426] Lustre: DEBUG MARKER: done [ 1239.974498] Lustre: server umount lustre-OST0000 complete [ 1247.204423] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1247.213518] LustreError: Skipped 1 previous similar message [ 1247.217277] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1247.230181] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1247.240372] Lustre: Skipped 1 previous similar message [ 1247.724424] LNetError: 120-3: Refusing connection from 192.168.201.151 for 127.0.0.2@tcp: No matching NI [ 1247.728589] LNetError: 47248:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1247.735071] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1248.736104] Lustre: 47260:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761317250/real 1761317250] req@00000000a75a3888 x1846874729394048/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1761317257 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 1249.759254] Lustre: 57224:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761317252/real 1761317252] req@00000000be2c0ff6 x1846874729395072/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761317258 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1249.980635] Lustre: server umount lustre-MDT0000 complete [ 1253.147485] LustreError: 56411:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761317262 with bad export cookie 5517662009532087242 [ 1253.156151] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1263.449974] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 1265.839930] Key type lgssc unregistered [ 1266.013058] LNet: 58026:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1266.026327] LNet: Removed LNI 192.168.201.151@tcp [ 1266.705029] Key type .llcrypt unregistered [ 1266.707052] Key type ._llcrypt unregistered [ 1292.530253] alg: No test for adler32 (adler32-zlib) [ 1293.281340] Key type ._llcrypt registered [ 1293.284683] Key type .llcrypt registered [ 1293.361643] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_hostid [ 1303.857182] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 1304.741175] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1305.098631] Lustre: Lustre: Build Version: 2.15.7_10_g040485f [ 1305.256959] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 1305.261885] LNet: Accept secure, port 988 [ 1306.903155] Key type lgssc registered [ 1307.726984] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1315.451125] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1321.202635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1326.469290] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1331.877705] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1336.932937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1336.989528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1338.212648] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1338.243922] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1338.319828] Lustre: lustre-MDT0000: new disk, initializing [ 1338.425793] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1338.446667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1341.547910] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 1350.031837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1350.100456] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1350.169941] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1350.203974] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1350.207034] Lustre: Skipped 1 previous similar message [ 1350.259953] Lustre: lustre-MDT0001: new disk, initializing [ 1350.336451] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1350.379492] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1350.390703] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1353.532965] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 1361.636421] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1366.342338] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1371.647992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1371.727341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1371.736661] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1371.901890] Lustre: lustre-OST0000: new disk, initializing [ 1371.908316] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1371.997643] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1375.618413] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 1377.296420] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1377.303836] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1383.766397] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1387.071542] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1387.250963] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1390.989910] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 1391.177233] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1392.610668] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1392.611964] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1392.618701] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1392.618905] Lustre: Skipped 1 previous similar message [ 1397.729834] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1397.740328] Lustre: Skipped 2 previous similar messages [ 1398.813359] Lustre: server umount lustre-OST0000 complete [ 1402.848934] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1402.870916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1402.874490] Lustre: Skipped 1 previous similar message [ 1407.975187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1408.515056] Lustre: server umount lustre-MDT0000 complete [ 1413.092891] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1413.127763] LustreError: Skipped 1 previous similar message [ 1413.480593] LustreError: 62986:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761317422 with bad export cookie 7150975856657721722 [ 1413.489250] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1413.496396] LustreError: 62986:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1413.770540] Lustre: server umount lustre-MDT0001 complete [ 1422.555681] Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 10:50:29 (1761317429) [ 1424.088614] Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST [ 1425.809824] Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 10:50:32 (1761317432) [ 1434.960727] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 1437.886940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1438.170289] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1438.245683] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1439.158163] Lustre: Failing over lustre-MDT0000 [ 1439.172208] LustreError: 67140:0:(osp_object.c:629:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1439.181630] LustreError: 67140:0:(lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1439.192750] LustreError: 67140:0:(lod_dev.c:424:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000 get update log failed: rc = -5 [ 1444.831163] Lustre: 67191:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761317448/real 1761317448] req@0000000074e9a51e x1846875063864576/t0(0) o251->MGC192.168.201.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761317454 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1445.014610] Lustre: server umount lustre-MDT0000 complete [ 1450.677155] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 10:50:57 (1761317457) [ 1456.275129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1456.737636] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1456.785864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1460.877657] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 1462.241443] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1467.364330] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1467.742203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1468.055218] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1471.417457] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 1478.547904] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1484.061597] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1490.696850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1491.010874] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1495.040927] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 1501.780819] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1511.167560] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1517.539719] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1517.554639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1517.564334] Lustre: Skipped 3 previous similar messages [ 1517.574907] Lustre: Skipped 3 previous similar messages [ 1518.903387] Lustre: server umount lustre-MDT0000 complete [ 1522.640300] LustreError: 67739:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761317531 with bad export cookie 7150975856657722849 [ 1522.650350] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1522.653611] LustreError: 67739:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1522.688360] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1522.696812] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1526.117591] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1530.599120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1538.572402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1539.049111] Lustre: Evicted from MGS (at 192.168.201.151@tcp) after server handle changed from 0x633d5ea7757a6de1 to 0x633d5ea7757a7574 [ 1539.065291] Lustre: MGC192.168.201.151@tcp: Connection restored to 192.168.201.151@tcp (at 0@lo) [ 1539.397163] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1539.476258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1543.189470] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 1549.656852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1549.797166] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1549.801097] Lustre: Skipped 1 previous similar message [ 1549.996778] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:65 [ 1552.749900] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 1555.432182] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 1558.453368] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1560.551775] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 1563.471791] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1570.274276] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1570.284033] LustreError: Skipped 1 previous similar message [ 1570.286095] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1570.300543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1570.476926] Lustre: server umount lustre-MDT0000 complete [ 1570.489738] Lustre: Skipped 1 previous similar message [ 1573.489988] LustreError: 67739:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761317582 with bad export cookie 7150975856657724788 [ 1573.495253] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1578.576323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1582.950911] Lustre: 60347:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761317585/real 1761317585] req@0000000053dc89e9 x1846875063894208/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761317592 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1582.969154] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1582.977031] Lustre: Skipped 2 previous similar messages [ 1585.405322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1586.143672] Lustre: 60348:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761317588/real 1761317588] req@00000000296db73a x1846875063894272/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761317595 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1588.191366] Lustre: 60345:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761317590/real 1761317590] req@00000000a4ff65b9 x1846875063894400/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761317597 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1590.177607] Lustre: Evicted from MGS (at 192.168.201.151@tcp) after server handle changed from 0x633d5ea7757a7574 to 0x633d5ea7757a7b86 [ 1590.187474] Lustre: MGC192.168.201.151@tcp: Connection restored to (at 0@lo) [ 1590.414937] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1590.424553] LustreError: Skipped 4 previous similar messages [ 1590.487306] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 1593.423691] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 1599.671116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1599.977187] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:97 [ 1600.998913] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 1603.143283] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 1608.411409] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1612.631844] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1618.401868] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1618.403041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1618.405859] Lustre: Skipped 2 previous similar messages [ 1618.408584] Lustre: Skipped 1 previous similar message [ 1619.180969] Lustre: server umount lustre-MDT0000 complete [ 1619.183167] Lustre: Skipped 1 previous similar message [ 1621.634542] LustreError: 70710:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761317630 with bad export cookie 7150975856657726342 [ 1621.636365] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1621.647875] LustreError: 70710:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1626.231652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1628.414640] Lustre: DEBUG MARKER: files compared the same [ 1635.807928] Lustre: 76883:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761317638/real 1761317638] req@0000000008f1331e x1846875063906304/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761317644 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1647.897378] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 1650.041379] Key type lgssc unregistered [ 1650.265560] LNet: 77793:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1650.277161] LNet: Removed LNI 192.168.201.151@tcp [ 1650.864156] Key type .llcrypt unregistered [ 1650.865954] Key type ._llcrypt unregistered [ 1667.757450] alg: No test for adler32 (adler32-zlib) [ 1668.510348] Key type ._llcrypt registered [ 1668.513144] Key type .llcrypt registered [ 1668.607823] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 1669.259137] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1669.494524] Lustre: Lustre: Build Version: 2.15.7_10_g040485f [ 1669.616315] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 1669.619854] LNet: Accept secure, port 988 [ 1671.255176] Key type lgssc registered [ 1672.002829] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1677.925342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1679.361213] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1679.407596] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1683.164900] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug +malloc all 8 [ 1684.453141] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1689.430343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1689.602334] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1692.889616] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug +malloc all 8 [ 1699.232236] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1703.977806] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1711.289932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1711.769763] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1715.514418] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug +malloc all 8 [ 1719.978561] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:129 [ 1722.821303] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1730.212776] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 1740.767836] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1740.769983] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1740.771705] LustreError: Skipped 1 previous similar message [ 1740.780322] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1740.782985] Lustre: Skipped 1 previous similar message [ 1744.381586] Lustre: server umount lustre-OST0000 complete [ 1751.010284] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1751.012720] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1751.020061] Lustre: Skipped 1 previous similar message [ 1751.021190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1752.799264] Lustre: server umount lustre-MDT0000 complete [ 1755.187623] LustreError: 79236:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761317764 with bad export cookie 8176367981985240678 [ 1755.188890] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1755.196357] LustreError: 79236:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1755.385799] Lustre: server umount lustre-MDT0001 complete [ 1763.284833] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 1765.329488] Key type lgssc unregistered [ 1765.524550] LNet: 82929:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1765.530214] LNet: Removed LNI 192.168.201.151@tcp [ 1766.163399] Key type .llcrypt unregistered [ 1766.169349] Key type ._llcrypt unregistered [ 1784.080692] alg: No test for adler32 (adler32-zlib) [ 1784.833378] Key type ._llcrypt registered [ 1784.835661] Key type .llcrypt registered [ 1784.920247] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 1785.520722] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1786.003764] Lustre: Lustre: Build Version: 2.15.7_10_g040485f [ 1786.230354] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 1786.238597] LNet: Accept secure, port 988 [ 1787.927135] Key type lgssc registered [ 1788.673975] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1794.874325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1812.511499] LustreError: 84364:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.201.151@tcp: failed processing log, type 1: rc = -5 [ 1843.231227] LustreError: 84364:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.201.151@tcp: failed processing log, type 4: rc = -110 [ 1873.951199] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1877.341894] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 1884.184881] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1891.383888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1891.446062] Lustre: *** cfs_fail_loc=706, val=0*** [ 1895.688740] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1895.747544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1895.796041] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:161 [ 1898.700264] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 1901.027549] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1904.529772] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1904.782856] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1904.842265] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:161 [ 1907.826981] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 1913.853090] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1918.192424] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1920.480967] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1920.485842] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1920.508324] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1921.503890] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1921.510426] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1921.520913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1923.553255] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1923.558033] Lustre: Skipped 1 previous similar message [ 1926.400466] Lustre: server umount lustre-OST0000 complete [ 1930.208623] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1930.213957] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1930.222826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1933.795073] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1935.469440] Lustre: server umount lustre-MDT0000 complete [ 1938.310189] LustreError: 84371:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761317947 with bad export cookie 12112220782360988013 [ 1938.314128] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1938.317672] LustreError: 84371:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1938.527253] Lustre: server umount lustre-MDT0001 complete [ 1945.679098] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 1947.839683] Key type lgssc unregistered [ 1948.013069] LNet: 88102:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1948.019372] LNet: Removed LNI 192.168.201.151@tcp [ 1948.519337] Key type .llcrypt unregistered [ 1948.521190] Key type ._llcrypt unregistered [ 1965.694623] alg: No test for adler32 (adler32-zlib) [ 1966.451502] Key type ._llcrypt registered [ 1966.453124] Key type .llcrypt registered [ 1966.577929] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 1967.201692] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1967.487947] Lustre: Lustre: Build Version: 2.15.7_10_g040485f [ 1967.647596] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 1967.651271] LNet: Accept secure, port 988 [ 1969.279295] Key type lgssc registered [ 1970.034788] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1976.719965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1982.308748] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 1989.843342] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1990.223064] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1990.312289] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1994.004896] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 1995.746273] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1995.754327] LustreError: Skipped 2 previous similar messages [ 2000.865266] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2000.877121] LustreError: Skipped 1 previous similar message [ 2001.065053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2001.371920] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2004.679513] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 2005.477953] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2005.483918] LustreError: Skipped 3 previous similar messages [ 2005.500245] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:193 [ 2010.497228] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2010.505952] LustreError: Skipped 3 previous similar messages [ 2010.695316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2010.764172] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:193 [ 2013.696057] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 2025.953996] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2025.955357] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2025.959433] LustreError: Skipped 1 previous similar message [ 2025.966874] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2025.972059] Lustre: Skipped 1 previous similar message [ 2031.053474] Lustre: server umount lustre-OST0000 complete [ 2036.192887] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2036.193417] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2036.194761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2036.211545] Lustre: Skipped 2 previous similar messages [ 2039.862645] Lustre: server umount lustre-MDT0000 complete [ 2041.314591] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2041.324273] LustreError: Skipped 2 previous similar messages [ 2042.896783] Lustre: server umount lustre-MDT0001 complete [ 2051.039386] Lustre: 92969:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761318054/real 1761318054] req@00000000195d3c95 x1846875758014848/t0(0) o251->MGC192.168.201.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761318060 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2051.080873] Lustre: server umount MGS complete [ 2059.201706] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 11:01:06 (1761318066) [ 2082.155778] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_hostid [ 2089.657720] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 2097.819872] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2104.544635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2110.718334] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2115.837337] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2120.774404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2120.836990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2123.885393] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 2130.704864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2130.752864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2130.906451] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2130.943214] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 2130.998202] Lustre: lustre-MDT0001: new disk, initializing [ 2131.061278] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2131.086168] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2131.090899] LustreError: 97683:0:(lod_dev.c:424:lod_sub_recovery_thread()) lustre-MDT0001-osd get update log failed: rc = -115 [ 2133.771743] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 2140.939183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2140.989109] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2140.992198] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2141.116747] Lustre: lustre-OST0000: new disk, initializing [ 2141.122764] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2141.176686] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2143.651935] Lustre: 98620:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@00000000faf23af8 x1846875758019328/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1761318164 ref 2 fl Rpc:RQU/2/0 rc 0/-115 job:'' [ 2143.736825] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 2150.006925] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2154.452690] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2154.490579] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2154.545503] Lustre: lustre-MDT0000: new disk, initializing [ 2154.656246] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2154.679535] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2155.747583] Lustre: 99532:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@00000000678a24cf x1846875758022784/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1761318176 ref 2 fl Rpc:RQU/2/0 rc 0/-115 job:'' [ 2157.496613] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 2157.792882] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2164.208213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2164.215234] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2164.899392] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 2184.672080] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2184.680256] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2184.688262] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2184.691287] Lustre: Skipped 1 previous similar message [ 2187.748481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2187.750598] Lustre: Skipped 1 previous similar message [ 2190.436754] Lustre: server umount lustre-OST0000 complete [ 2197.984292] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2197.986190] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2197.988269] LustreError: Skipped 1 previous similar message [ 2197.988808] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2197.988812] Lustre: Skipped 1 previous similar message [ 2198.028780] Lustre: Skipped 2 previous similar messages [ 2199.604417] Lustre: server umount lustre-MDT0000 complete [ 2216.927200] Lustre: 100727:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761318220/real 1761318220] req@00000000b212f2cb x1846875758033024/t0(0) o251->MGC192.168.201.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761318226 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2216.974398] Lustre: server umount MGS complete [ 2216.976279] Lustre: Skipped 1 previous similar message [ 2227.521772] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 11:03:54 (1761318234) [ 2243.840828] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 2246.242844] Key type lgssc unregistered [ 2246.489924] LNet: 102136:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2246.497526] LNet: Removed LNI 192.168.201.151@tcp [ 2247.182324] Key type .llcrypt unregistered [ 2247.184900] Key type ._llcrypt unregistered [ 2259.548440] alg: No test for adler32 (adler32-zlib) [ 2260.307440] Key type ._llcrypt registered [ 2260.309828] Key type .llcrypt registered [ 2260.395296] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 2261.100665] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2261.808742] Lustre: Lustre: Build Version: 2.15.7_10_g040485f [ 2262.066718] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 2262.071646] LNet: Accept secure, port 988 [ 2263.767139] Key type lgssc registered [ 2264.640450] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2270.444755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2271.817058] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2271.854975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2274.989363] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 2277.344927] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2280.946226] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2281.217624] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2284.269478] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 2290.093656] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2294.076716] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2298.966438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2299.187541] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2302.008857] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 2302.254614] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 2307.171806] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2316.771137] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2316.778086] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2316.779585] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2316.792385] Lustre: Skipped 1 previous similar message [ 2321.895293] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2321.900530] Lustre: Skipped 1 previous similar message [ 2322.069446] Lustre: server umount lustre-OST0000 complete [ 2327.010491] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2327.011040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2327.018532] Lustre: Skipped 2 previous similar messages [ 2330.659624] Lustre: server umount lustre-MDT0000 complete [ 2332.129586] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2332.140223] LustreError: Skipped 1 previous similar message [ 2333.314792] LustreError: 103400:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761318342 with bad export cookie 16153468361280198376 [ 2333.320403] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2333.322279] LustreError: 103400:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2333.567889] Lustre: server umount lustre-MDT0001 complete [ 2342.095409] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 2344.159412] Key type lgssc unregistered [ 2344.349959] LNet: 107072:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2344.358789] LNet: Removed LNI 192.168.201.151@tcp [ 2344.941462] Key type .llcrypt unregistered [ 2344.948314] Key type ._llcrypt unregistered [ 2355.808979] alg: No test for adler32 (adler32-zlib) [ 2356.563427] Key type ._llcrypt registered [ 2356.564783] Key type .llcrypt registered [ 2356.662586] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 2357.266687] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2357.527840] Lustre: Lustre: Build Version: 2.15.7_10_g040485f [ 2357.675557] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 2357.680808] LNet: Accept secure, port 988 [ 2359.327127] Key type lgssc registered [ 2360.252743] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2364.644846] LustreError: 108182:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 716 sleeping [ 2364.738542] LustreError: 108203:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 716 waking [ 2364.742229] LustreError: 108182:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 716 awake: rc=4901 [ 2364.745619] LustreError: 108203:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2364.750761] LustreError: 108203:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2364.762363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2366.052156] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2366.081202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2371.482794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2371.552715] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2371.722800] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2375.127533] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 2380.077611] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing lsmod [ 2382.408924] LustreError: 109508:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 716 sleeping [ 2382.420252] LustreError: 109511:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 716 waking [ 2382.423993] LustreError: 109508:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 716 awake: rc=4993 [ 2382.428490] LustreError: 109508:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2382.438928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2382.444302] LustreError: 109508:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2382.800586] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2386.916865] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2386.920226] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2386.929930] Lustre: Skipped 1 previous similar message [ 2386.945838] Lustre: Skipped 2 previous similar messages [ 2391.280781] Lustre: server umount lustre-MDT0000 complete [ 2392.037584] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2392.047396] LustreError: Skipped 2 previous similar messages [ 2394.486680] LustreError: 108220:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761318403 with bad export cookie 14715316855674378548 [ 2394.493800] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2394.502164] LustreError: 108220:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2394.834609] Lustre: server umount lustre-MDT0001 complete [ 2403.807193] Lustre: 110104:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761318406/real 1761318406] req@00000000f4ad408e x1846876166955904/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761318412 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2404.035905] Lustre: server umount lustre-OST0000 complete [ 2411.433974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2411.714651] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2411.790978] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2414.720180] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 2417.120760] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2420.515771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2420.779893] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2423.412362] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 2428.013178] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2431.854886] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2436.330087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2436.568967] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2439.013521] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 2442.740073] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:65 [ 2443.937881] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2461.152903] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2461.154854] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2461.155302] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2461.157162] Lustre: Skipped 2 previous similar messages [ 2464.590532] Lustre: server umount lustre-OST0000 complete [ 2471.394188] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2471.396984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2471.406282] Lustre: Skipped 1 previous similar message [ 2471.410187] Lustre: Skipped 1 previous similar message [ 2473.191993] Lustre: server umount lustre-MDT0000 complete [ 2475.462962] LustreError: 110560:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761318484 with bad export cookie 14715316855674379234 [ 2475.465726] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2475.468700] LustreError: 110560:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2482.615438] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 2484.201256] Key type lgssc unregistered [ 2484.388231] LNet: 114251:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2484.394273] LNet: Removed LNI 192.168.201.151@tcp [ 2484.886922] Key type .llcrypt unregistered [ 2484.888333] Key type ._llcrypt unregistered [ 2497.951897] alg: No test for adler32 (adler32-zlib) [ 2498.703860] Key type ._llcrypt registered [ 2498.706198] Key type .llcrypt registered [ 2498.758534] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 2499.351721] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2499.592677] Lustre: Lustre: Build Version: 2.15.7_10_g040485f [ 2499.732440] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 2499.736568] LNet: Accept secure, port 988 [ 2501.351736] Key type lgssc registered [ 2502.151636] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2508.455833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2509.813479] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2509.859423] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2513.309485] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 2514.913616] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2519.878727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2520.108265] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2523.053099] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 2528.680637] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2533.038177] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2538.179577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2538.455112] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2541.557410] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 2541.740578] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 2548.595242] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2556.739911] Lustre: Setting parameter lustre-client.llite.some_wrong_param in log lustre-client [ 2559.294869] Lustre: Modifying parameter lustre-client.llite.some_wrong_param in log lustre-client [ 2564.285923] Lustre: server umount lustre-OST0000 complete [ 2567.138623] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2567.140644] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2567.147170] Lustre: Skipped 1 previous similar message [ 2567.154641] Lustre: Skipped 1 previous similar message [ 2572.261994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2573.279483] Lustre: 118674:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761318576/real 1761318576] req@00000000cf45f95c x1846876315860288/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761318582 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2573.477696] Lustre: server umount lustre-MDT0000 complete [ 2574.308064] Lustre: 115026:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761318576/real 1761318576] req@00000000d44c34b6 x1846876315860544/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761318583 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2574.315030] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2574.332772] Lustre: 115026:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2574.369662] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2574.381054] LustreError: Skipped 1 previous similar message [ 2576.363105] LustreError: 118321:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761318585 with bad export cookie 13959500949823549122 [ 2576.364426] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2576.373476] LustreError: 118321:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2576.591147] Lustre: server umount lustre-MDT0001 complete [ 2585.216016] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 2587.737251] Key type lgssc unregistered [ 2587.976819] LNet: 119478:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2587.991516] LNet: Removed LNI 192.168.201.151@tcp [ 2588.657554] Key type .llcrypt unregistered [ 2588.665679] Key type ._llcrypt unregistered [ 2602.415977] alg: No test for adler32 (adler32-zlib) [ 2603.182115] Key type ._llcrypt registered [ 2603.184062] Key type .llcrypt registered [ 2603.264391] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 2604.533545] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2605.110024] Lustre: Lustre: Build Version: 2.15.7_10_g040485f [ 2605.346691] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 2605.351478] LNet: Accept secure, port 988 [ 2606.997297] Key type lgssc registered [ 2608.015970] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2617.689366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2619.181254] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2619.261830] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2623.563821] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 2624.485538] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2629.601964] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2631.287787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2631.740903] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2635.975889] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 2642.768930] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2647.520207] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2653.291715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2653.531107] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2657.537024] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 2664.302571] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2669.876924] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 2680.290793] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2680.293249] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2680.305829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2680.325829] Lustre: Skipped 1 previous similar message [ 2683.645974] Lustre: server umount lustre-OST0000 complete [ 2687.967947] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2687.976098] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2688.005990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2688.008570] Lustre: Skipped 1 previous similar message [ 2690.529412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2690.532957] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2693.912433] Lustre: server umount lustre-MDT0000 complete [ 2695.649990] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2695.660480] LustreError: Skipped 1 previous similar message [ 2697.283221] LustreError: 120727:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761318706 with bad export cookie 9408247824691766497 [ 2697.288368] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2697.628690] Lustre: server umount lustre-MDT0001 complete [ 2708.059510] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 2710.525958] Key type lgssc unregistered [ 2710.773366] LNet: 124427:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2710.790697] LNet: Removed LNI 192.168.201.151@tcp [ 2711.495445] Key type .llcrypt unregistered [ 2711.497964] Key type ._llcrypt unregistered [ 2730.550868] alg: No test for adler32 (adler32-zlib) [ 2731.305566] Key type ._llcrypt registered [ 2731.308556] Key type .llcrypt registered [ 2731.407813] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 2732.235361] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2732.728773] Lustre: Lustre: Build Version: 2.15.7_10_g040485f [ 2732.886732] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 2732.889996] LNet: Accept secure, port 988 [ 2734.535134] Key type lgssc registered [ 2735.602902] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2743.291524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2744.821895] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2744.890877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2749.296975] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 2749.922370] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2755.046565] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2756.541159] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2756.864782] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2760.165273] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 2766.135874] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2770.286720] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2775.349440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2775.522588] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2775.629917] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2779.153241] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 2780.652556] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 2785.876677] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2792.242624] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash in log lustre-MDT0000 [ 2795.938758] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids in log lustre-MDT0000 [ 2795.942078] Lustre: Skipped 2 previous similar messages [ 2801.082071] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash in log lustre-MDT0000 [ 2801.084538] Lustre: Skipped 2 previous similar messages [ 2802.841474] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 2802.899488] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2809.181857] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids in log lustre-MDT0000 [ 2809.190072] Lustre: Skipped 2 previous similar messages [ 2812.838778] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.51@tcp 192.168.0.[2,10]@tcp [ 2821.099643] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2821.103323] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2821.112227] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2821.604733] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2821.619092] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2821.633572] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2824.790353] Lustre: server umount lustre-OST0000 complete [ 2828.770244] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2828.781317] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2828.791767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2831.840319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2831.843922] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2834.680855] Lustre: server umount lustre-MDT0000 complete [ 2836.961615] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2836.973196] LustreError: Skipped 1 previous similar message [ 2837.860275] LustreError: 125870:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761318847 with bad export cookie 5707515103677042733 [ 2837.863164] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2837.873469] LustreError: 125870:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2838.223630] Lustre: server umount lustre-MDT0001 complete [ 2848.115354] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 2849.991699] Key type lgssc unregistered [ 2850.194577] LNet: 130495:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2850.202858] LNet: Removed LNI 192.168.201.151@tcp [ 2850.769758] Key type .llcrypt unregistered [ 2850.774911] Key type ._llcrypt unregistered [ 2871.661630] alg: No test for adler32 (adler32-zlib) [ 2872.416342] Key type ._llcrypt registered [ 2872.417982] Key type .llcrypt registered [ 2872.512720] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 2873.314253] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2873.858899] Lustre: Lustre: Build Version: 2.15.7_10_g040485f [ 2874.036355] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 2874.039907] LNet: Accept secure, port 988 [ 2875.719126] Key type lgssc registered [ 2876.615555] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2883.360424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2884.696510] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2884.745620] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2884.768057] Lustre: lustre-MDT0000: nosquash_nids set to 2@gni 0@lo 192.168.201.51@tcp 192.168.0.[2,10]@tcp [ 2884.781691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2888.529481] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 2890.211828] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2894.584966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2894.755781] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2894.771333] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.51@tcp 192.168.0.[2,10]@tcp [ 2894.805790] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2897.262285] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 2903.087490] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2908.246836] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2914.311622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2914.602292] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2915.661910] Lustre: lustre-OST0000: deleting orphan objects from 0x0:167 to 0x0:193 [ 2918.366089] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 2924.858904] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2935.271494] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2935.277412] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2935.278830] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2935.285610] Lustre: Skipped 1 previous similar message [ 2938.369922] Lustre: server umount lustre-OST0000 complete [ 2940.897423] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2940.901425] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2940.911465] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2940.914487] Lustre: Skipped 1 previous similar message [ 2945.506488] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2945.508163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2945.520965] Lustre: Skipped 1 previous similar message [ 2950.624055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2955.231127] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2955.369771] Lustre: server umount lustre-MDT0000 complete [ 2956.771145] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2956.797831] LustreError: Skipped 1 previous similar message [ 2958.529160] LustreError: 134058:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761318967 with bad export cookie 6055546625685777967 [ 2958.529564] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2958.542587] LustreError: 134058:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2958.770659] Lustre: server umount lustre-MDT0001 complete [ 2968.226530] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 2970.508048] Key type lgssc unregistered [ 2970.756333] LNet: 135810:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2970.761551] LNet: Removed LNI 192.168.201.151@tcp [ 2971.340769] Key type .llcrypt unregistered [ 2971.342371] Key type ._llcrypt unregistered [ 2993.674320] alg: No test for adler32 (adler32-zlib) [ 2994.426445] Key type ._llcrypt registered [ 2994.428206] Key type .llcrypt registered [ 2994.495791] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_hostid [ 3004.693279] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 3005.359948] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3005.668095] Lustre: Lustre: Build Version: 2.15.7_10_g040485f [ 3005.843409] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 3005.846554] LNet: Accept secure, port 988 [ 3007.487124] Key type lgssc registered [ 3008.219765] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3014.528033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3018.663392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3022.371824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3026.615749] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3031.054467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3031.103641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3032.274365] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3032.292342] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3032.344136] Lustre: lustre-MDT0000: new disk, initializing [ 3032.402543] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3032.412880] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3034.897605] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 3041.681993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3041.731521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3041.775501] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3041.802547] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3041.806250] Lustre: Skipped 1 previous similar message [ 3041.856836] Lustre: lustre-MDT0001: new disk, initializing [ 3041.922498] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3041.945963] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3041.953310] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3044.514107] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 3052.172274] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3057.168484] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3062.143538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3062.228540] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3062.237253] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3062.373465] Lustre: lustre-OST0000: new disk, initializing [ 3062.380400] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3062.466946] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3065.503517] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 3071.516155] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3071.523634] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3072.385462] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3075.487935] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3075.630793] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3078.662728] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3078.808836] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3081.695808] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3081.700914] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3081.707570] LustreError: Skipped 1 previous similar message [ 3081.713946] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3081.718734] Lustre: Skipped 1 previous similar message [ 3086.140258] Lustre: server umount lustre-OST0000 complete [ 3091.944747] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3091.948743] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3091.960616] Lustre: Skipped 2 previous similar messages [ 3091.972120] Lustre: Skipped 1 previous similar message [ 3094.483473] Lustre: server umount lustre-MDT0000 complete [ 3096.858983] LustreError: 139324:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761319106 with bad export cookie 1608194068727259607 [ 3096.864373] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3096.866041] LustreError: 139324:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3097.048986] Lustre: server umount lustre-MDT0001 complete [ 3102.074474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3102.328887] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3102.376858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3105.005786] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 3107.809502] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3110.343855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3110.566031] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3112.820570] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 3117.042359] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3120.270802] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3124.046861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3124.213493] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3126.452273] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 3131.546832] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3134.503717] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3134.632363] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3137.831050] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3137.963811] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3144.090488] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3144.144502] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3144.150677] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 3144.256252] Lustre: lustre-OST0001: new disk, initializing [ 3144.261480] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 3144.331235] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3147.628606] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 3152.928684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3152.934063] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3153.416343] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 3153.555203] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3156.308928] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 3156.445707] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3160.161825] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff965608477000.ost_server_uuid 40 [ 3161.219132] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff965608477000.ost_server_uuid in FULL state after 0 sec [ 3168.228258] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3168.230769] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3168.231951] LustreError: Skipped 1 previous similar message [ 3168.241334] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3168.243816] Lustre: Skipped 1 previous similar message [ 3172.035720] Lustre: server umount lustre-OST0001 complete [ 3173.347496] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3173.358603] LustreError: Skipped 1 previous similar message [ 3174.370859] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3174.374483] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3174.379934] Lustre: Skipped 1 previous similar message [ 3174.383453] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3176.417290] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3178.463994] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3178.465643] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3178.466917] Lustre: Skipped 2 previous similar messages [ 3178.474035] LustreError: Skipped 1 previous similar message [ 3180.379289] Lustre: server umount lustre-OST0000 complete [ 3183.584804] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3183.594965] Lustre: Skipped 2 previous similar messages [ 3188.706447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3188.712477] Lustre: Skipped 3 previous similar messages [ 3189.036095] Lustre: server umount lustre-MDT0000 complete [ 3191.048637] LustreError: 147221:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761319200 with bad export cookie 1608194068727260468 [ 3191.051844] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3191.054830] LustreError: 147221:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3202.962684] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 3204.839892] Key type lgssc unregistered [ 3205.072550] LNet: 150329:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3205.078750] LNet: Removed LNI 192.168.201.151@tcp [ 3205.668606] Key type .llcrypt unregistered [ 3205.670752] Key type ._llcrypt unregistered [ 3227.435856] alg: No test for adler32 (adler32-zlib) [ 3228.189440] Key type ._llcrypt registered [ 3228.190711] Key type .llcrypt registered [ 3228.259427] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_hostid [ 3238.149639] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 3238.798599] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3238.996798] Lustre: Lustre: Build Version: 2.15.7_10_g040485f [ 3239.144800] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 3239.150149] LNet: Accept secure, port 988 [ 3240.767138] Key type lgssc registered [ 3241.392243] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3246.991222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3251.060697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3254.829469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3258.925553] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3263.282404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3263.332245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3264.499690] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3264.526807] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3264.594709] Lustre: lustre-MDT0000: new disk, initializing [ 3264.660610] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3264.697786] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3267.373440] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 3274.740768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3274.793843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3274.848069] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3274.885833] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3274.889691] Lustre: Skipped 1 previous similar message [ 3274.955281] Lustre: lustre-MDT0001: new disk, initializing [ 3275.028437] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3275.055899] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3275.067139] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3278.088666] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 3284.081292] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3286.985277] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3290.236524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3290.274531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3290.278749] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3290.407900] Lustre: lustre-OST0000: new disk, initializing [ 3290.411286] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3290.455345] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3292.558177] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 3297.816221] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3297.820971] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3298.150266] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3300.391842] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3300.504938] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3302.877695] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3303.003576] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3308.002116] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3308.003327] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3308.007921] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3308.012051] Lustre: Skipped 1 previous similar message [ 3310.372351] Lustre: server umount lustre-OST0000 complete [ 3316.193884] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3316.200371] LustreError: Skipped 1 previous similar message [ 3316.202666] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3316.213447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3316.216179] Lustre: Skipped 1 previous similar message [ 3318.241053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3318.241213] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3319.465172] Lustre: server umount lustre-MDT0000 complete [ 3321.859187] LustreError: 154417:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761319331 with bad export cookie 3726873539424574738 [ 3321.863218] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3321.869085] LustreError: 154417:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3322.045257] Lustre: server umount lustre-MDT0001 complete [ 3327.287791] Lustre: DEBUG MARKER: == conf-sanity test 47: server restart does not make client loss lru_resize settings ========================================================== 11:22:14 (1761319334) [ 3334.790273] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_hostid [ 3338.334638] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 3343.576955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3346.827972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3349.630391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3352.975457] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3357.078498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3357.122289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3357.272309] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3357.288713] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3357.358592] Lustre: lustre-MDT0000: new disk, initializing [ 3357.428582] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3357.444766] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3360.227871] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 3367.208470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3367.267970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3367.341855] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3367.373900] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3367.378742] Lustre: Skipped 1 previous similar message [ 3367.382315] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3367.384914] Lustre: Skipped 1 previous similar message [ 3367.429377] Lustre: lustre-MDT0001: new disk, initializing [ 3367.492834] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3367.555207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3367.566321] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3369.830875] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 3376.108257] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3379.343273] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3383.210611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3383.249950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3383.254666] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3383.393698] Lustre: lustre-OST0000: new disk, initializing [ 3383.396346] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3383.448426] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3385.861679] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 3391.472389] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3393.546467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3393.552846] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3399.489337] Lustre: Failing over lustre-OST0000 [ 3399.587317] Lustre: server umount lustre-OST0000 complete [ 3408.863230] Lustre: 152645:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761319411/real 1761319411] req@00000000cc327f06 x1846877091821376/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1761319418 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 3408.878099] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3409.951218] Lustre: 152645:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761319412/real 1761319412] req@00000000b8ff3918 x1846877091821952/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1761319419 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 3410.975158] Lustre: 152646:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761319412/real 1761319412] req@00000000086786c9 x1846877091821824/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761319419 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3410.990022] Lustre: 152646:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3415.277523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3415.461996] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3415.478549] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3416.032140] Lustre: 152647:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761319418/real 1761319418] req@00000000a9411f65 x1846877091822336/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761319425 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3416.045284] Lustre: 152647:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3416.411435] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3417.794771] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 3419.639294] Lustre: Failing over lustre-MDT0000 [ 3419.760539] Lustre: server umount lustre-MDT0000 complete [ 3420.512794] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3420.516152] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3420.518975] Lustre: Skipped 2 previous similar messages [ 3421.536126] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3421.541614] LustreError: Skipped 4 previous similar messages [ 3426.651161] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.201.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3426.657755] LustreError: Skipped 1 previous similar message [ 3427.807174] Lustre: 152647:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761319429/real 1761319429] req@00000000a641736a x1846877091826112/t0(0) o400->MGC192.168.201.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761319436 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3427.819674] Lustre: 152647:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3427.824507] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3428.718780] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.201.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3428.725731] LustreError: Skipped 6 previous similar messages [ 3432.928601] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3432.937484] LustreError: Skipped 2 previous similar messages [ 3434.552248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3444.061189] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.201.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3444.072877] LustreError: Skipped 8 previous similar messages [ 3445.218909] Lustre: Evicted from MGS (at 192.168.201.151@tcp) after server handle changed from 0x33b883bd00346068 to 0x33b883bd00346523 [ 3445.226906] Lustre: MGC192.168.201.151@tcp: Connection restored to (at 0@lo) [ 3445.334651] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 3445.386046] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3447.695279] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 3450.205075] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3450.854247] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 3450.868541] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 3450.897448] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 3455.968345] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3455.972328] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3455.977832] Lustre: Skipped 1 previous similar message [ 3455.980543] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3455.983111] Lustre: Skipped 1 previous similar message [ 3458.665937] Lustre: server umount lustre-OST0000 complete [ 3461.088757] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3461.094097] LustreError: Skipped 1 previous similar message [ 3466.208694] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3466.211578] Lustre: Skipped 3 previous similar messages [ 3468.614536] LustreError: 161396:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761319477 with bad export cookie 3726873539424576803 [ 3468.616986] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3468.619464] LustreError: 161396:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3474.726560] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 3476.234894] Key type lgssc unregistered [ 3476.393669] LNet: 167248:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3476.401284] LNet: Removed LNI 192.168.201.151@tcp [ 3476.845523] Key type .llcrypt unregistered [ 3476.847285] Key type ._llcrypt unregistered [ 3488.988632] alg: No test for adler32 (adler32-zlib) [ 3489.741420] Key type ._llcrypt registered [ 3489.743311] Key type .llcrypt registered [ 3489.820985] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 3490.249996] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3490.535255] Lustre: Lustre: Build Version: 2.15.7_10_g040485f [ 3490.637721] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 3490.640474] LNet: Accept secure, port 988 [ 3492.247109] Key type lgssc registered [ 3492.723356] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3496.896659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3498.124550] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3498.170570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3500.690246] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 3503.584978] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3505.268582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3505.479646] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3507.795239] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 3511.904650] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3515.205500] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3518.894268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3519.125879] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3521.358695] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 3525.455417] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3528.369342] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:65 [ 3542.892409] Lustre: lustre-MDT0000: Client 987f7f1d-8759-4534-bf7b-b3ea06c88649 (at 192.168.201.51@tcp) reconnecting [ 3578.738155] Lustre: lustre-MDT0000: Client 987f7f1d-8759-4534-bf7b-b3ea06c88649 (at 192.168.201.51@tcp) reconnecting [ 3657.597948] Lustre: lustre-MDT0000: Client 987f7f1d-8759-4534-bf7b-b3ea06c88649 (at 192.168.201.51@tcp) reconnecting [ 3702.666109] Lustre: lustre-MDT0000: Client 987f7f1d-8759-4534-bf7b-b3ea06c88649 (at 192.168.201.51@tcp) reconnecting [ 3751.802070] Lustre: lustre-MDT0000: Client 987f7f1d-8759-4534-bf7b-b3ea06c88649 (at 192.168.201.51@tcp) reconnecting [ 3800.959216] Lustre: lustre-MDT0000: Client 987f7f1d-8759-4534-bf7b-b3ea06c88649 (at 192.168.201.51@tcp) reconnecting [ 3854.208648] Lustre: lustre-MDT0000: Client 987f7f1d-8759-4534-bf7b-b3ea06c88649 (at 192.168.201.51@tcp) reconnecting [ 3907.482467] Lustre: lustre-MDT0000: Client 987f7f1d-8759-4534-bf7b-b3ea06c88649 (at 192.168.201.51@tcp) reconnecting [ 4014.001328] Lustre: lustre-MDT0000: Client 987f7f1d-8759-4534-bf7b-b3ea06c88649 (at 192.168.201.51@tcp) reconnecting [ 4014.009090] Lustre: Skipped 1 previous similar message [ 4194.224174] Lustre: lustre-MDT0000: Client 987f7f1d-8759-4534-bf7b-b3ea06c88649 (at 192.168.201.51@tcp) reconnecting [ 4194.234064] Lustre: Skipped 3 previous similar messages [ 4401.613171] Lustre: Failing over lustre-MDT0000 [ 4401.636749] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4401.642549] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4401.660118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4401.789610] Lustre: server umount lustre-MDT0000 complete [ 4403.045303] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.201.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4404.196238] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4404.198754] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4404.208701] LustreError: Skipped 1 previous similar message [ 4404.224926] Lustre: Skipped 1 previous similar message [ 4408.161652] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.201.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4408.170860] LustreError: Skipped 1 previous similar message [ 4411.360737] Lustre: 168020:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761320413/real 1761320413] req@000000006f6b61eb x1846877355420096/t0(0) o400->MGC192.168.201.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761320420 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4411.392598] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4411.403713] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4411.415046] LustreError: Skipped 6 previous similar messages [ 4416.352646] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.201.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4416.358152] LustreError: Skipped 1 previous similar message [ 4418.908222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4426.592160] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.201.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4426.601496] LustreError: Skipped 10 previous similar messages [ 4428.776040] Lustre: Evicted from MGS (at 192.168.201.151@tcp) after server handle changed from 0xac9968fe8627d2a6 to 0xac9968fe862a3ec7 [ 4428.792325] Lustre: MGC192.168.201.151@tcp: Connection restored to 192.168.201.151@tcp (at 0@lo) [ 4429.006847] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4429.067646] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4432.664909] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 4432.739329] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4434.405763] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 4434.416295] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 4434.452705] Lustre: lustre-OST0000: deleting orphan objects from 0x0:68 to 0x0:97 [ 4440.146742] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4441.575255] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4460.001853] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4460.003499] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4460.013337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4460.029626] Lustre: Skipped 2 previous similar messages [ 4462.897738] Lustre: server umount lustre-MDT0000 complete [ 4465.122191] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4465.130032] LustreError: Skipped 9 previous similar messages [ 4465.635655] LustreError: 168688:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761320474 with bad export cookie 12437087288342232775 [ 4465.642216] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4465.645711] LustreError: 168688:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4470.246146] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4470.262677] Lustre: Skipped 2 previous similar messages [ 4470.270692] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4472.085932] Lustre: server umount lustre-MDT0001 complete [ 4481.330280] Lustre: server umount lustre-OST0000 complete [ 4487.855950] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_hostid [ 4493.359543] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 4499.842413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4504.611792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4508.780931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4512.887131] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4517.490234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4517.536278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4517.702968] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4517.729191] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4517.778954] Lustre: lustre-MDT0000: new disk, initializing [ 4517.834736] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4517.850830] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4521.198618] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 4529.528230] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4529.583372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4529.641856] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4529.680520] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 4529.685088] Lustre: Skipped 1 previous similar message [ 4529.744518] Lustre: lustre-MDT0001: new disk, initializing [ 4529.799463] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4529.827369] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4529.838250] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4533.010337] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 4540.362093] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4544.471584] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4548.895923] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4548.956348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4548.965275] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4549.120447] Lustre: lustre-OST0000: new disk, initializing [ 4549.125438] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 4549.180749] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4552.084771] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 4555.288767] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4555.298771] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4558.726355] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4561.680093] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 4561.819283] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4564.603899] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 4564.759222] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4570.593865] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4570.594765] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4570.597420] LustreError: Skipped 1 previous similar message [ 4570.603307] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4570.610093] Lustre: Skipped 2 previous similar messages [ 4575.714530] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4575.720107] Lustre: Skipped 1 previous similar message [ 4580.319167] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 4580.415801] Lustre: server umount lustre-OST0000 complete [ 4585.953378] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4585.954336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4585.964542] Lustre: Skipped 2 previous similar messages [ 4585.975826] Lustre: Skipped 1 previous similar message [ 4597.218530] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 4597.308626] Lustre: server umount lustre-MDT0000 complete [ 4599.766250] LustreError: 175686:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761320608 with bad export cookie 12437087288342233489 [ 4599.770180] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4599.773230] LustreError: 175686:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4605.095852] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 11:43:32 (1761320612) [ 4613.964887] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_hostid [ 4619.635329] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 4625.696811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4630.016557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4634.057096] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4638.106969] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4642.007251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4642.043622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4642.163323] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4642.178132] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4642.224555] Lustre: lustre-MDT0000: new disk, initializing [ 4642.271058] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4642.285369] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4644.939164] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 4651.166176] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4651.215691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4651.280682] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4651.305171] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 4651.308102] Lustre: Skipped 1 previous similar message [ 4651.354186] Lustre: lustre-MDT0001: new disk, initializing [ 4651.419480] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4651.476895] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4651.486515] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4653.892761] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 4659.917292] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4663.224292] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4667.182158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4667.245222] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4667.250298] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4667.391238] Lustre: lustre-OST0000: new disk, initializing [ 4667.398535] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 4667.444715] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4669.916421] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 4675.093726] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4675.098550] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4675.558987] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4680.164666] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4680.171262] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4680.174051] Lustre: Skipped 6 previous similar messages [ 4680.174655] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4685.235811] Lustre: server umount lustre-OST0000 complete [ 4685.237588] Lustre: Skipped 1 previous similar message [ 4695.523307] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4695.532952] LustreError: Skipped 3 previous similar messages [ 4696.064161] LustreError: 184764:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761320705 with bad export cookie 12437087288342234343 [ 4696.069137] LustreError: 184764:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4696.069393] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4702.055443] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 11:45:09 (1761320709) [ 4711.747204] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_hostid [ 4716.891214] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 4722.145870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4725.969451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4729.905555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4733.549506] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4737.205521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4737.259545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4737.407973] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4737.433627] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4737.485902] Lustre: lustre-MDT0000: new disk, initializing [ 4737.555579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4737.568384] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4739.983820] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 4746.682188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4746.724557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4746.807338] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4746.979039] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4749.505403] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 4755.615728] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4758.489275] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4761.829818] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4761.878159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4761.886139] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4764.291382] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 4768.775327] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4769.780598] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4773.859889] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4773.860616] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4773.863113] LustreError: Skipped 2 previous similar messages [ 4773.870884] Lustre: Skipped 3 previous similar messages [ 4773.875589] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4773.879584] Lustre: Skipped 3 previous similar messages [ 4775.170673] Lustre: server umount lustre-OST0000 complete [ 4775.172963] Lustre: Skipped 2 previous similar messages [ 4784.096687] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4784.103162] LustreError: Skipped 1 previous similar message [ 4785.526889] LustreError: 192006:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761320794 with bad export cookie 12437087288342235421 [ 4785.527688] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4785.535567] LustreError: 192006:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4792.691727] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 4794.129298] Key type lgssc unregistered [ 4794.273650] LNet: 193095:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4794.278264] LNet: Removed LNI 192.168.201.151@tcp [ 4794.713824] Key type .llcrypt unregistered [ 4794.716988] Key type ._llcrypt unregistered [ 4808.025885] alg: No test for adler32 (adler32-zlib) [ 4808.782569] Key type ._llcrypt registered [ 4808.785432] Key type .llcrypt registered [ 4808.851070] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 4809.320920] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4809.604920] Lustre: Lustre: Build Version: 2.15.7_10_g040485f [ 4809.712101] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 4809.714500] LNet: Accept secure, port 988 [ 4811.303124] Key type lgssc registered [ 4811.837631] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4815.761688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4816.972804] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4817.010226] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4819.265711] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 4822.499991] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4824.560270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4824.755583] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4827.026909] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 4831.120343] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4834.338839] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4838.100416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4838.276265] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4840.424856] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 4844.166107] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4847.528606] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 4862.944802] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4862.950040] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4862.952354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4862.962488] Lustre: Skipped 1 previous similar message [ 4868.070099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4868.076403] Lustre: Skipped 2 previous similar messages [ 4868.777908] Lustre: server umount lustre-OST0000 complete [ 4871.138415] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4871.141537] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4871.153816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4873.186624] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4873.186777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4873.195478] Lustre: Skipped 1 previous similar message [ 4876.955258] Lustre: server umount lustre-MDT0000 complete [ 4878.306704] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4878.315248] LustreError: Skipped 1 previous similar message [ 4878.833700] LustreError: 194547:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761320888 with bad export cookie 7337492032733795795 [ 4878.838785] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4878.841280] LustreError: 194547:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4879.002394] Lustre: server umount lustre-MDT0001 complete [ 4885.144627] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 4886.379198] Key type lgssc unregistered [ 4886.513407] LNet: 198236:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4886.519868] LNet: Removed LNI 192.168.201.151@tcp [ 4886.894963] Key type .llcrypt unregistered [ 4886.896634] Key type ._llcrypt unregistered [ 4898.865356] alg: No test for adler32 (adler32-zlib) [ 4899.618448] Key type ._llcrypt registered [ 4899.620901] Key type .llcrypt registered [ 4899.674029] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 4900.218042] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4900.384320] Lustre: Lustre: Build Version: 2.15.7_10_g040485f [ 4900.484088] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 4900.486480] LNet: Accept secure, port 988 [ 4902.095140] Key type lgssc registered [ 4902.572407] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4906.797796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4908.031645] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4908.057798] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4910.115814] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 4913.122751] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4914.605572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4914.787350] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4916.794537] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 4920.138749] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4922.725264] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4925.872770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4926.039136] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4927.942553] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 4931.519786] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4938.341332] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 4945.888670] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4945.893408] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4945.901713] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4947.305542] Lustre: lustre-OST0000: Not available for connect from 192.168.201.51@tcp (stopping) [ 4948.962816] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4948.963646] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4948.969477] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4952.411595] Lustre: lustre-OST0000: Not available for connect from 192.168.201.51@tcp (stopping) [ 4952.415478] Lustre: Skipped 1 previous similar message [ 4957.534250] Lustre: lustre-OST0000: Not available for connect from 192.168.201.51@tcp (stopping) [ 4957.539256] Lustre: Skipped 2 previous similar messages [ 4959.200436] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 4959.286910] Lustre: server umount lustre-OST0000 complete [ 4963.745210] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9656180a8800.ost_server_uuid 40 [ 4964.700109] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 4976.096299] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4976.100149] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4976.105920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4976.108610] Lustre: Skipped 2 previous similar messages [ 4979.681336] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4981.603835] Lustre: server umount lustre-MDT0000 complete [ 4983.400473] LustreError: 201650:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761320992 with bad export cookie 847564609631280430 [ 4983.404136] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4983.406215] LustreError: 201650:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4983.545496] Lustre: server umount lustre-MDT0001 complete [ 4987.236520] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 11:49:54 (1761320994) [ 4990.824895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4991.058221] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4991.094183] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4992.986644] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 4996.580026] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4996.980338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4997.186803] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4999.143341] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 5002.509303] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5004.910328] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5007.918401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5008.085072] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5010.038694] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 5013.735562] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5017.239059] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5017.282859] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5017.288132] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 5017.365301] Lustre: lustre-OST0001: new disk, initializing [ 5017.369135] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 5019.566499] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 5021.543335] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 5023.255081] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5023.260736] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5024.434812] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5026.784090] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5026.788558] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5026.798207] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5026.800985] Lustre: Skipped 2 previous similar messages [ 5027.831743] Lustre: server umount lustre-OST0000 complete [ 5028.322684] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5028.330803] LustreError: Skipped 1 previous similar message [ 5030.775744] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 5030.884456] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5032.983477] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 5033.087406] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5038.562479] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5038.569016] LustreError: Skipped 5 previous similar messages [ 5048.802954] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5048.804894] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5048.812114] Lustre: Skipped 1 previous similar message [ 5048.815504] LustreError: Skipped 1 previous similar message [ 5049.897900] Lustre: server umount lustre-OST0001 complete [ 5059.040071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5059.042918] Lustre: Skipped 6 previous similar messages [ 5066.207129] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 5066.261271] Lustre: server umount lustre-MDT0000 complete [ 5067.921308] LustreError: 203668:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761321077 with bad export cookie 847564609631281494 [ 5067.923257] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5067.926109] LustreError: 203668:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5070.836623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5072.794978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5074.767631] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5076.914777] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5080.457831] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 11:51:27 (1761321087) [ 5083.644192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5083.846902] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5083.866631] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5083.978534] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5083.982028] Lustre: Skipped 1 previous similar message [ 5085.804444] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 5089.506082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5089.576901] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5089.596338] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5091.636916] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 5095.144605] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5098.054364] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5101.402615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5101.504237] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5102.190768] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:129 [ 5103.763180] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 5107.109319] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5110.105390] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5110.143576] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5112.219283] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 5115.922486] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5115.942486] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3 to 0x0:33 [ 5117.409280] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5117.418088] LustreError: Skipped 2 previous similar messages [ 5117.419135] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5117.423612] Lustre: Skipped 2 previous similar messages [ 5123.326925] Lustre: server umount lustre-OST0000 complete [ 5123.330482] Lustre: Skipped 1 previous similar message [ 5126.117307] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5126.123571] LustreError: Skipped 6 previous similar messages [ 5136.355190] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5136.357159] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5136.358689] LustreError: Skipped 2 previous similar messages [ 5136.363084] Lustre: Skipped 10 previous similar messages [ 5157.345856] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 5157.401245] Lustre: server umount lustre-MDT0000 complete [ 5157.403336] Lustre: Skipped 1 previous similar message [ 5159.133207] LustreError: 210077:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761321168 with bad export cookie 847564609631282859 [ 5159.137148] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5159.137886] LustreError: 210077:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5161.702932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5163.288912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5165.098931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5167.062937] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5170.074285] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 11:52:57 (1761321177) [ 5178.249939] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_hostid [ 5182.514639] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 5187.119497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5190.385402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5193.017808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5195.812661] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5199.439446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5199.484589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5199.600715] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5199.621550] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5199.665852] Lustre: lustre-MDT0000: new disk, initializing [ 5199.710185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5199.714764] Lustre: Skipped 3 previous similar messages [ 5199.726076] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5201.750041] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 5207.484343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5207.523776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5207.569480] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5207.595467] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 5207.597773] Lustre: Skipped 1 previous similar message [ 5207.641299] Lustre: lustre-MDT0001: new disk, initializing [ 5207.715746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5207.725045] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5209.691865] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 5214.479335] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5217.098891] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5220.235286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5220.272883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5220.276774] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 5220.404726] Lustre: lustre-OST0000: new disk, initializing [ 5220.407755] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5222.353805] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 5227.632126] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5228.546295] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5228.550751] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5229.848896] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 5229.945565] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5231.868519] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 5231.974887] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5233.634735] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5233.641533] LustreError: Skipped 1 previous similar message [ 5233.643529] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5233.650242] Lustre: Skipped 5 previous similar messages [ 5234.981375] Lustre: server umount lustre-OST0000 complete [ 5234.983242] Lustre: Skipped 1 previous similar message [ 5243.874599] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5243.880369] LustreError: Skipped 9 previous similar messages [ 5244.561055] LustreError: 220368:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761321253 with bad export cookie 847564609631284581 [ 5244.564400] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5244.564719] LustreError: 220368:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5248.294099] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5250.305017] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 5253.937361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5255.986372] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 5259.739485] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5262.326974] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5265.414524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5265.567293] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5265.571203] Lustre: Skipped 4 previous similar messages [ 5267.446301] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 5270.574966] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5272.490471] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 5278.744828] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 6 sec [ 5280.610350] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 5280.704241] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5283.301339] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5283.303967] Lustre: Skipped 11 previous similar messages [ 5290.184187] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 5291.915338] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 5316.074597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5317.811820] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 5321.138137] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5322.830464] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 5335.520849] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5335.527491] Lustre: Skipped 5 previous similar messages [ 5346.271102] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 5346.783663] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5346.791225] LustreError: Skipped 3 previous similar messages [ 5347.874354] LustreError: 222621:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761321357 with bad export cookie 847564609631285435 [ 5347.877050] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5347.879263] LustreError: 222621:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5351.450333] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 11:55:59 (1761321359) [ 5354.063990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5356.031541] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 5359.397347] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5361.322926] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 5364.483307] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5367.098597] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5370.011081] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5371.746502] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 5374.702966] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5376.442859] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 5384.749523] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec [ 5386.494637] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 5386.587794] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5389.285222] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5389.319174] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5389.322808] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 5389.382396] Lustre: lustre-OST0001: new disk, initializing [ 5389.384766] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 5391.222349] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 5395.728573] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5397.510516] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5397.515563] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5397.552128] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 5397.644930] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5399.400273] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 5399.490980] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5402.595228] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5402.598285] LustreError: Skipped 4 previous similar messages [ 5406.316380] Lustre: server umount lustre-OST0001 complete [ 5406.318534] Lustre: Skipped 6 previous similar messages [ 5408.807409] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 5408.894629] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5410.617509] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 5410.716280] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5434.777598] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5434.867598] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5434.871244] Lustre: Skipped 5 previous similar messages [ 5436.592797] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 5439.542412] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5439.975697] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5439.980856] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5439.983394] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to (at 0@lo) [ 5439.988902] Lustre: Skipped 1 previous similar message [ 5440.278222] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5470.175127] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 5472.224173] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5472.228957] Lustre: Skipped 7 previous similar messages [ 5479.650452] LustreError: 228847:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761321488 with bad export cookie 847564609631286373 [ 5479.651683] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5479.655802] LustreError: 228847:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5488.596334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5490.355460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5492.173551] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5494.019255] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5496.884740] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 11:58:24 (1761321504) [ 5499.995055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5500.137478] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5500.154711] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5502.150103] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 5505.811737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5505.867656] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5507.764165] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 5511.253286] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5513.682405] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5516.629573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5516.725101] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5518.680039] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 5521.798085] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5531.678087] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5531.726608] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5533.607460] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 5536.684778] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5538.377164] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 5540.128364] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 5540.230842] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5541.983710] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 5542.082224] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5544.383398] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff96560682c000.ost_server_uuid 40 [ 5545.041810] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff96560682c000.ost_server_uuid in FULL state after 0 sec [ 5565.952623] Lustre: Permanently deactivating lustre-OST0001 [ 5565.958932] Lustre: Setting parameter lustre-OST0001-osc.osc.active in log lustre-client [ 5565.961871] Lustre: Skipped 1 previous similar message [ 5566.838583] Lustre: Permanently reactivating lustre-OST0001 [ 5568.996705] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5568.997156] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5569.000381] LustreError: Skipped 5 previous similar messages [ 5569.002566] Lustre: Skipped 18 previous similar messages [ 5582.303181] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 5584.353908] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5584.362840] LustreError: Skipped 29 previous similar messages [ 5606.907072] LustreError: 237826:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761321616 with bad export cookie 847564609631287633 [ 5606.911122] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5606.911362] LustreError: 237826:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5609.050173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5610.543094] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5612.046642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5613.432941] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5615.726926] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 12:00:23 (1761321623) [ 5616.108921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5618.485306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5618.595978] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5618.608817] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5618.612154] Lustre: Skipped 5 previous similar messages [ 5620.262421] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 5623.347762] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5623.384194] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5625.009571] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 5627.892266] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5630.113730] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5632.538699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5632.567671] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5632.649328] Lustre: Permanently deactivating lustre-OST0000 [ 5634.308744] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 5636.857498] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5639.200533] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5640.826282] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 5642.378210] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5643.660986] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5644.223430] Lustre: Permanently reactivating lustre-OST0000 [ 5651.452070] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 5651.455405] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5651.461229] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to (at 0@lo) [ 5651.476986] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 5651.485775] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 5675.999101] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 5676.003869] Lustre: Skipped 1 previous similar message [ 5676.067811] Lustre: server umount lustre-OST0000 complete [ 5676.070041] Lustre: Skipped 9 previous similar messages [ 5685.306047] LustreError: 244689:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761321694 with bad export cookie 847564609631289222 [ 5685.308836] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5685.313619] LustreError: 244689:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5690.322538] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 5691.332484] Key type lgssc unregistered [ 5691.450424] LNet: 249638:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5691.454713] LNet: Removed LNI 192.168.201.151@tcp [ 5691.751774] Key type .llcrypt unregistered [ 5691.753514] Key type ._llcrypt unregistered [ 5700.771434] alg: No test for adler32 (adler32-zlib) [ 5701.522335] Key type ._llcrypt registered [ 5701.523327] Key type .llcrypt registered [ 5701.563902] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 5701.935384] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5702.189150] Lustre: Lustre: Build Version: 2.15.7_10_g040485f [ 5702.279839] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 5702.282883] LNet: Accept secure, port 988 [ 5703.879119] Key type lgssc registered [ 5704.264532] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5705.511075] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5707.880575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5709.046646] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5709.070179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5710.619201] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 5713.574410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5713.603537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5713.639082] Lustre: Found index 1 for lustre-MDT0001, updating log [ 5713.650516] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5713.653765] Lustre: Permanently deactivating lustre-MDT0001 [ 5713.758293] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5714.606344] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 5714.609948] LustreError: 251191:0:(osp_object.c:629:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5714.614495] LustreError: 251191:0:(lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5714.618620] LustreError: 251191:0:(lod_dev.c:424:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000 get update log failed: rc = -108 [ 5715.272093] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 5717.983188] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5719.961193] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5722.443293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5722.552929] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5724.050961] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 5724.578652] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5724.582184] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:161 [ 5726.757553] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5729.254191] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5729.329855] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5730.346120] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:131 to 0x280000400:161 [ 5730.350597] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:99 to 0x2c0000400:129 [ 5730.803150] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 5733.372525] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5734.995709] Lustre: Permanently reactivating lustre-MDT0001 [ 5735.001016] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active in log lustre-client [ 5735.004504] Lustre: Skipped 3 previous similar messages [ 5742.433293] Lustre: lustre-MDT0001: Client d2965bfe-f035-43e3-84df-3f06a59af03d (at 192.168.201.51@tcp) reconnecting [ 5745.116663] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5745.737957] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5747.578100] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 40 [ 5747.657616] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 5749.298443] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 40 [ 5749.374368] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 5750.298643] Lustre: Permanently deactivating lustre-MDT0001 [ 5750.307355] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active in log lustre-client [ 5750.310583] Lustre: Skipped 2 previous similar messages [ 5760.058275] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 5770.209541] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5770.224085] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5770.239069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5772.257848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5772.259717] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5772.274931] Lustre: Skipped 2 previous similar messages [ 5772.298417] Lustre: Skipped 2 previous similar messages [ 5773.445192] Lustre: server umount lustre-MDT0000 complete [ 5777.358364] LustreError: 251140:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761321786 with bad export cookie 5417900534987307470 [ 5777.366359] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5777.385906] LustreError: 251140:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5777.397329] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5777.406669] LustreError: Skipped 1 previous similar message [ 5777.722831] Lustre: server umount lustre-MDT0001 complete [ 5784.070415] Lustre: server umount lustre-OST0000 complete [ 5794.006755] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 12:03:21 (1761321801) [ 5799.689148] Lustre: server umount lustre-OST0001 complete [ 5802.783451] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_hostid [ 5806.554217] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 5810.738914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5813.470712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5816.382131] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5819.249448] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5822.176919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5822.218389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5822.346197] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5822.349717] Lustre: Skipped 2 previous similar messages [ 5822.370412] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5822.418705] Lustre: lustre-MDT0000: new disk, initializing [ 5822.468831] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5822.477655] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5824.592512] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 5830.213854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5830.256590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5830.301221] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5830.331983] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 5830.334594] Lustre: Skipped 1 previous similar message [ 5830.377775] Lustre: lustre-MDT0001: new disk, initializing [ 5830.450353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5830.455754] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5832.520181] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 5837.652460] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5840.447396] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5843.831640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5843.870740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5843.874468] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 5843.991515] Lustre: lustre-OST0000: new disk, initializing [ 5843.994094] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5844.038161] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5844.041530] Lustre: Skipped 1 previous similar message [ 5846.059427] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 5847.635663] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5847.641329] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5851.144302] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5851.994318] LustreError: 259688:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 sleeping for 10000ms [ 5856.883159] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5856.921672] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5856.926525] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 5857.026954] Lustre: lustre-OST0001: new disk, initializing [ 5857.029400] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 5858.943834] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 5862.096093] LustreError: 259688:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 awake [ 5862.106251] LustreError: 259688:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 sleeping for 10000ms [ 5863.390855] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5863.962623] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5863.967176] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5864.145054] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5872.199106] LustreError: 259688:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 awake [ 5872.996626] Lustre: lustre-OST0001: Not available for connect from 192.168.201.51@tcp (stopping) [ 5873.000055] Lustre: Skipped 1 previous similar message [ 5874.146617] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5874.149211] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5878.108210] Lustre: lustre-OST0001: Not available for connect from 192.168.201.51@tcp (stopping) [ 5878.111044] Lustre: Skipped 2 previous similar messages [ 5879.023102] Lustre: server umount lustre-OST0001 complete [ 5879.266286] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5879.272352] LustreError: Skipped 1 previous similar message [ 5884.387471] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5884.393733] LustreError: Skipped 1 previous similar message [ 5894.112212] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5894.112758] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5894.115687] LustreError: Skipped 2 previous similar messages [ 5894.120127] Lustre: Skipped 1 previous similar message [ 5894.122702] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5894.126685] Lustre: Skipped 1 previous similar message [ 5894.625662] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5894.629969] LustreError: Skipped 3 previous similar messages [ 5897.760187] Lustre: server umount lustre-OST0000 complete [ 5899.744644] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5899.751279] Lustre: Skipped 1 previous similar message [ 5904.864744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5904.869531] Lustre: Skipped 3 previous similar messages [ 5907.230223] LustreError: 259674:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761321916 with bad export cookie 5417900534987309983 [ 5907.232741] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5907.235910] LustreError: 259674:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5918.802760] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 5919.955093] Key type lgssc unregistered [ 5920.082464] LNet: 264876:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5920.087401] LNet: Removed LNI 192.168.201.151@tcp [ 5920.437120] Key type .llcrypt unregistered [ 5920.438374] Key type ._llcrypt unregistered [ 5935.088214] alg: No test for adler32 (adler32-zlib) [ 5935.840442] Key type ._llcrypt registered [ 5935.841681] Key type .llcrypt registered [ 5935.887448] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_hostid [ 5943.153223] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 5943.659701] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5943.814733] Lustre: Lustre: Build Version: 2.15.7_10_g040485f [ 5943.905471] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 5943.908917] LNet: Accept secure, port 988 [ 5945.495167] Key type lgssc registered [ 5945.964822] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5950.372870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5953.651788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5956.862250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5959.721772] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5963.053502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5963.088084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5964.199120] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5964.216306] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5964.258353] Lustre: lustre-MDT0000: new disk, initializing [ 5964.313523] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5964.322636] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5966.181589] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 5971.270269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5971.299461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5971.341343] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5971.365929] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 5971.368316] Lustre: Skipped 1 previous similar message [ 5971.407530] Lustre: lustre-MDT0001: new disk, initializing [ 5971.455573] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5971.478812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5971.487426] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5973.310880] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 5977.990681] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5980.595797] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5983.748443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5983.792834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5983.796906] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 5983.912712] Lustre: lustre-OST0000: new disk, initializing [ 5983.915639] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5983.955467] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5985.964576] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 5990.928657] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5992.984910] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5992.989889] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5993.299244] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 5993.403430] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5995.646737] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 5995.759696] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5998.048952] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5998.049968] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5998.051889] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5998.057803] Lustre: Skipped 1 previous similar message [ 5998.060798] LustreError: Skipped 1 previous similar message [ 6002.784050] Lustre: server umount lustre-OST0000 complete [ 6007.265233] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6007.268841] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6007.275771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6007.278776] Lustre: Skipped 1 previous similar message [ 6008.288096] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6008.288770] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6010.457329] Lustre: server umount lustre-MDT0000 complete [ 6012.226521] LustreError: 269817:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761322021 with bad export cookie 2547295097924153807 [ 6012.229736] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6012.234137] LustreError: 269817:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6012.377695] Lustre: server umount lustre-MDT0001 complete [ 6015.931748] Lustre: DEBUG MARKER: == conf-sanity test 52: check recovering objects from lost+found ========================================================== 12:07:03 (1761322023) [ 6018.735591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6018.937496] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6018.964184] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6020.771663] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 6024.160472] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6024.568222] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6024.715608] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6026.519052] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 6029.949404] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6032.621739] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6035.741420] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6035.928881] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6037.990739] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 6041.446402] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6059.488614] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6059.490265] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6059.493225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6059.498630] Lustre: Skipped 1 previous similar message [ 6059.501337] Lustre: Skipped 1 previous similar message [ 6060.900175] Lustre: server umount lustre-OST0000 complete [ 6062.660656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 6067.952970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6068.105473] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6069.780780] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 6072.899259] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6075.301917] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6075.306826] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to (at 0@lo) [ 6075.306849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6075.310238] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:65 [ 6075.317755] Lustre: lustre-OST0000: deleting orphan objects from 0x0:42 to 0x0:65 [ 6080.481400] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6080.481691] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6080.487647] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6080.490604] LustreError: Skipped 1 previous similar message [ 6080.493499] Lustre: Skipped 1 previous similar message [ 6082.920497] Lustre: server umount lustre-OST0000 complete [ 6090.720102] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6090.726639] LustreError: Skipped 1 previous similar message [ 6092.297557] LustreError: 273497:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761322101 with bad export cookie 2547295097924154647 [ 6092.301767] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6092.301909] LustreError: 273497:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6092.447040] Lustre: server umount lustre-MDT0001 complete [ 6092.449106] Lustre: Skipped 1 previous similar message [ 6097.626761] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 6098.766692] Key type lgssc unregistered [ 6098.901920] LNet: 278947:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6098.907066] LNet: Removed LNI 192.168.201.151@tcp [ 6099.241682] Key type .llcrypt unregistered [ 6099.243465] Key type ._llcrypt unregistered [ 6109.582627] alg: No test for adler32 (adler32-zlib) [ 6110.335360] Key type ._llcrypt registered [ 6110.336831] Key type .llcrypt registered [ 6110.377180] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 6110.734741] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6110.988536] Lustre: Lustre: Build Version: 2.15.7_10_g040485f [ 6111.078314] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 6111.081138] LNet: Accept secure, port 988 [ 6112.671142] Key type lgssc registered [ 6113.089536] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6116.242586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6117.421787] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6117.440118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6119.186928] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 6122.464347] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6122.642025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6122.786753] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6124.506104] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 6127.669442] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6130.117088] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6132.842871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6132.983405] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6134.755124] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 6138.050831] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6147.303383] Lustre: lustre-OST0000: deleting orphan objects from 0x0:42 to 0x0:97 [ 6158.816528] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6158.820470] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6158.826760] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6162.912662] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6162.912796] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6162.913891] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6163.938678] Lustre: server umount lustre-OST0000 complete [ 6168.032291] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6168.039200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6168.042104] Lustre: Skipped 2 previous similar messages [ 6171.730476] Lustre: server umount lustre-MDT0000 complete [ 6173.152374] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6173.158712] LustreError: Skipped 1 previous similar message [ 6173.290212] LustreError: 281194:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761322182 with bad export cookie 12174913573694581996 [ 6173.293330] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6173.298093] LustreError: 281194:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6173.425763] Lustre: server umount lustre-MDT0001 complete [ 6178.966056] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 6180.140581] Key type lgssc unregistered [ 6180.269571] LNet: 284855:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6180.274335] LNet: Removed LNI 192.168.201.151@tcp [ 6180.633553] Key type .llcrypt unregistered [ 6180.635166] Key type ._llcrypt unregistered [ 6188.260851] alg: No test for adler32 (adler32-zlib) [ 6189.012403] Key type ._llcrypt registered [ 6189.013756] Key type .llcrypt registered [ 6189.060465] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 6189.543580] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6189.695242] Lustre: Lustre: Build Version: 2.15.7_10_g040485f [ 6189.797231] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 6189.798877] LNet: Accept secure, port 988 [ 6191.383122] Key type lgssc registered [ 6191.770677] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6194.710500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6195.871177] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6195.888911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6197.575894] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 6200.798226] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6200.923683] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6202.501395] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 6205.374859] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6207.510946] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6210.004676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6210.138444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6211.174994] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 6211.827205] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 6214.675966] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6226.914029] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6226.914210] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6226.916069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6226.919966] Lustre: Skipped 1 previous similar message [ 6226.923524] LustreError: Skipped 1 previous similar message [ 6230.060126] Lustre: server umount lustre-OST0000 complete [ 6232.035109] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6232.036456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6232.039515] Lustre: Skipped 1 previous similar message [ 6232.043919] Lustre: Skipped 2 previous similar messages [ 6237.152047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6237.154767] Lustre: Skipped 1 previous similar message [ 6242.272038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6242.275567] Lustre: Skipped 1 previous similar message [ 6246.367169] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 6246.446496] Lustre: server umount lustre-MDT0000 complete [ 6248.113260] LustreError: 289139:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761322257 with bad export cookie 8762067771442756853 [ 6248.116599] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6248.119045] LustreError: 289139:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6248.261526] Lustre: server umount lustre-MDT0001 complete [ 6253.764080] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 6254.916944] Key type lgssc unregistered [ 6255.031435] LNet: 289943:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6255.034855] LNet: Removed LNI 192.168.201.151@tcp [ 6255.356386] Key type .llcrypt unregistered [ 6255.358064] Key type ._llcrypt unregistered [ 6262.962965] alg: No test for adler32 (adler32-zlib) [ 6263.714466] Key type ._llcrypt registered [ 6263.715896] Key type .llcrypt registered [ 6263.758938] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 6264.213675] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6264.388160] Lustre: Lustre: Build Version: 2.15.7_10_g040485f [ 6264.476502] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 6264.478871] LNet: Accept secure, port 988 [ 6266.063125] Key type lgssc registered [ 6266.450432] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6269.388908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6270.547492] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6270.567677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6272.158201] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 6275.553446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6275.734148] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6277.465831] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 6280.792216] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6283.275918] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6286.145224] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6286.285143] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6288.080852] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 6291.410156] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6300.645983] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 6305.761092] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6305.761794] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6305.762559] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6305.766079] Lustre: Skipped 1 previous similar message [ 6307.038517] Lustre: server umount lustre-OST0000 complete [ 6310.880726] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6310.886938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6310.889877] Lustre: Skipped 2 previous similar messages [ 6314.636105] Lustre: server umount lustre-MDT0000 complete [ 6316.000886] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6316.007765] LustreError: Skipped 1 previous similar message [ 6316.230322] LustreError: 294079:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761322325 with bad export cookie 18177452163819362338 [ 6316.232662] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6316.236151] LustreError: 294079:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6316.372700] Lustre: server umount lustre-MDT0001 complete [ 6321.741556] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 6322.903484] Key type lgssc unregistered [ 6323.034371] LNet: 294881:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6323.040825] LNet: Removed LNI 192.168.201.151@tcp [ 6323.371624] Key type .llcrypt unregistered [ 6323.372840] Key type ._llcrypt unregistered [ 6333.891125] alg: No test for adler32 (adler32-zlib) [ 6334.643517] Key type ._llcrypt registered [ 6334.645110] Key type .llcrypt registered [ 6334.692951] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 6335.071529] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6335.318838] Lustre: Lustre: Build Version: 2.15.7_10_g040485f [ 6335.417931] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 6335.420376] LNet: Accept secure, port 988 [ 6337.007156] Key type lgssc registered [ 6337.433452] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6340.585632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6341.808463] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6341.831961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6343.571576] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 6347.232659] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6347.324771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6347.485095] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6349.416896] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 6352.908594] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6355.157929] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6357.871119] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6358.022176] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6359.707699] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 6362.773412] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6372.388747] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:193 [ 6382.561311] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6382.561863] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6382.562166] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6382.567729] Lustre: Skipped 1 previous similar message [ 6387.680340] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6387.684099] Lustre: Skipped 2 previous similar messages [ 6388.127368] Lustre: server umount lustre-OST0000 complete [ 6392.803497] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6392.805346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6392.808874] Lustre: Skipped 1 previous similar message [ 6392.814113] Lustre: Skipped 1 previous similar message [ 6395.714884] Lustre: server umount lustre-MDT0000 complete [ 6397.135398] LustreError: 296320:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761322406 with bad export cookie 118584730687608230 [ 6397.138453] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6397.140398] LustreError: 296320:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6397.267473] Lustre: server umount lustre-MDT0001 complete [ 6402.494614] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 6403.561804] Key type lgssc unregistered [ 6403.681363] LNet: 300834:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6403.685040] LNet: Removed LNI 192.168.201.151@tcp [ 6404.010861] Key type .llcrypt unregistered [ 6404.012663] Key type ._llcrypt unregistered [ 6411.474725] alg: No test for adler32 (adler32-zlib) [ 6412.226494] Key type ._llcrypt registered [ 6412.227872] Key type .llcrypt registered [ 6412.267788] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 6412.639803] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6412.875813] Lustre: Lustre: Build Version: 2.15.7_10_g040485f [ 6412.960399] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 6412.962847] LNet: Accept secure, port 988 [ 6414.543112] Key type lgssc registered [ 6414.921144] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6417.976537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6419.165201] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6419.183421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6420.848594] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 6424.263023] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6424.414199] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6426.061647] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 6428.876044] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6431.157390] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6433.903684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6434.028041] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6435.047761] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:225 [ 6435.714440] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 6438.849103] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6444.512886] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6444.512994] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6444.517969] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6444.521520] Lustre: Skipped 1 previous similar message [ 6444.525262] Lustre: Skipped 1 previous similar message [ 6449.632785] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6449.635508] Lustre: Skipped 1 previous similar message [ 6449.761908] Lustre: server umount lustre-OST0000 complete [ 6454.752287] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6454.752822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6454.756333] Lustre: Skipped 1 previous similar message [ 6457.410594] Lustre: server umount lustre-MDT0000 complete [ 6458.790505] LustreError: 302924:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761322468 with bad export cookie 12649056498205023955 [ 6458.793355] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6458.794888] LustreError: 302924:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6458.935839] Lustre: server umount lustre-MDT0001 complete [ 6463.763584] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 6464.803787] Key type lgssc unregistered [ 6464.929323] LNet: 305932:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6464.932564] LNet: Removed LNI 192.168.201.151@tcp [ 6465.258112] Key type .llcrypt unregistered [ 6465.259568] Key type ._llcrypt unregistered [ 6472.369645] alg: No test for adler32 (adler32-zlib) [ 6473.120447] Key type ._llcrypt registered [ 6473.121596] Key type .llcrypt registered [ 6473.161848] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 6473.493085] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6473.738968] Lustre: Lustre: Build Version: 2.15.7_10_g040485f [ 6473.827837] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 6473.830199] LNet: Accept secure, port 988 [ 6475.415135] Key type lgssc registered [ 6475.811685] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6478.778701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6479.938337] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6479.955984] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6481.508214] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 6484.787450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6484.931108] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6486.585186] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 6489.548390] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6491.708480] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6494.325485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6494.453108] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6496.021197] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 6496.486223] Lustre: lustre-OST0000: deleting orphan objects from 0x0:227 to 0x0:257 [ 6498.883644] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6506.976028] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6506.980044] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6506.986847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6511.073141] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6511.074636] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6511.078376] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6512.482688] Lustre: server umount lustre-OST0000 complete [ 6515.679895] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6515.683555] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6515.688623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6515.691460] Lustre: Skipped 1 previous similar message [ 6520.071461] Lustre: server umount lustre-MDT0000 complete [ 6521.312181] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6521.319864] LustreError: Skipped 1 previous similar message [ 6521.536985] LustreError: 307192:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761322530 with bad export cookie 16409996388311449098 [ 6521.539182] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6521.541454] LustreError: 307192:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6521.678108] Lustre: server umount lustre-MDT0001 complete [ 6526.762814] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 6527.830915] Key type lgssc unregistered [ 6527.960394] LNet: 310862:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6527.963777] LNet: Removed LNI 192.168.201.151@tcp [ 6528.281286] Key type .llcrypt unregistered [ 6528.282840] Key type ._llcrypt unregistered [ 6533.177711] alg: No test for adler32 (adler32-zlib) [ 6533.929308] Key type ._llcrypt registered [ 6533.930723] Key type .llcrypt registered [ 6533.968734] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 6540.484213] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_hostid [ 6546.016092] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 6546.358713] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6546.558424] Lustre: Lustre: Build Version: 2.15.7_10_g040485f [ 6546.651703] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 6546.653821] LNet: Accept secure, port 988 [ 6548.239115] Key type lgssc registered [ 6548.614769] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6551.954097] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6554.522201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6556.941638] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6559.410448] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6561.985821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6562.013250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6563.100585] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6563.113706] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 6563.154916] Lustre: lustre-MDT0000: new disk, initializing [ 6563.183042] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6563.191143] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6564.719427] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 6569.134208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6569.156310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6569.192496] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6569.210751] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 6569.213292] Lustre: Skipped 1 previous similar message [ 6569.251850] Lustre: lustre-MDT0001: new disk, initializing [ 6569.288699] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6569.301905] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6569.305542] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6570.973798] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 6575.161885] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6577.225040] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6579.768662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6579.793767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6579.797538] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 6579.893341] Lustre: lustre-OST0000: new disk, initializing [ 6579.895783] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 6579.928603] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6581.534243] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 6582.646323] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6582.650793] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6585.671629] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6587.335643] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 6587.422887] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6589.074162] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 6589.158462] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6592.995782] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6592.996241] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6592.997134] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6593.001334] LustreError: Skipped 1 previous similar message [ 6593.012058] Lustre: Skipped 1 previous similar message [ 6598.114071] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6598.116455] Lustre: Skipped 2 previous similar messages [ 6603.233711] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6603.236501] Lustre: Skipped 1 previous similar message [ 6604.255376] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 6604.336262] Lustre: server umount lustre-OST0000 complete [ 6608.353361] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6608.354122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6608.358726] Lustre: Skipped 1 previous similar message [ 6612.037260] Lustre: server umount lustre-MDT0000 complete [ 6613.472647] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6613.477980] LustreError: Skipped 1 previous similar message [ 6613.569213] LustreError: 318287:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761322622 with bad export cookie 16093760960859636451 [ 6613.571826] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6613.573680] LustreError: 318287:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6613.694705] Lustre: server umount lustre-MDT0001 complete [ 6616.872750] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 12:17:04 (1761322624) [ 6619.500509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6619.664781] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6619.687686] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6621.278963] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 6624.470483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6624.621162] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6626.242862] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 6628.969513] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6631.177491] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6633.676712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6633.824885] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6635.493423] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 6638.198860] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6644.192747] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6644.194050] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6644.194993] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6644.195020] Lustre: Skipped 1 previous similar message [ 6644.206721] Lustre: Skipped 1 previous similar message [ 6648.216291] Lustre: server umount lustre-OST0000 complete [ 6650.335820] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6650.338636] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6654.432355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6654.434267] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6654.437264] Lustre: Skipped 2 previous similar messages [ 6655.814926] Lustre: server umount lustre-MDT0000 complete [ 6657.544303] LustreError: 321722:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761322666 with bad export cookie 16093760960859637305 [ 6657.546973] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6657.548571] LustreError: 321722:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6662.787135] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 6663.808329] Key type lgssc unregistered [ 6663.918535] LNet: 322813:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6663.922859] LNet: Removed LNI 192.168.201.151@tcp [ 6664.253731] Key type .llcrypt unregistered [ 6664.255134] Key type ._llcrypt unregistered [ 6674.928742] alg: No test for adler32 (adler32-zlib) [ 6675.681403] Key type ._llcrypt registered [ 6675.683703] Key type .llcrypt registered [ 6675.731243] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_hostid [ 6681.645450] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 6681.996759] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6682.264469] Lustre: Lustre: Build Version: 2.15.7_10_g040485f [ 6682.361927] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 6682.364556] LNet: Accept secure, port 988 [ 6683.959137] Key type lgssc registered [ 6684.378023] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6687.885608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6690.299140] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6692.518222] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6694.758737] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6696.543169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6698.526152] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6701.143175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6701.166760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6702.250917] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6702.261863] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 6702.299544] Lustre: lustre-MDT0000: new disk, initializing [ 6702.326120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6702.334346] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6703.990438] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 6708.485992] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6708.520354] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6708.557420] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6708.577298] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 6708.580107] Lustre: Skipped 1 previous similar message [ 6708.619098] Lustre: lustre-MDT0001: new disk, initializing [ 6708.660258] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6708.676425] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6708.681758] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6710.309707] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 6714.334676] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6716.380836] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6718.760454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6718.782183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6718.785142] Lustre: lustre-OST2710-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 6718.871377] Lustre: lustre-OST2710: new disk, initializing [ 6718.874373] Lustre: srv-lustre-OST2710: No data found on store. Initialize space: rc = -61 [ 6718.903613] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 6720.372468] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 6724.200835] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 6726.138605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 6726.144106] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 6726.561728] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6726.587549] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6726.590421] Lustre: lustre-OST03e8-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 6726.640470] Lustre: lustre-OST03e8: new disk, initializing [ 6726.643223] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space: rc = -61 [ 6726.679175] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 6728.206987] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 6730.970076] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 6733.900482] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 40 [ 6733.990705] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6735.519959] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 40 [ 6735.601426] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6736.379755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 6736.383903] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 6737.144067] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 40 [ 6737.219223] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6738.738022] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 40 [ 6738.820272] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6741.473354] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6741.474776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6741.478717] Lustre: Skipped 3 previous similar messages [ 6741.480429] Lustre: Skipped 2 previous similar messages [ 6745.989629] Lustre: server umount lustre-MDT0000 complete [ 6746.593757] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6746.599057] LustreError: Skipped 3 previous similar messages [ 6747.283958] LustreError: 327603:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761322756 with bad export cookie 575677082180776665 [ 6747.286919] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6747.290032] LustreError: 327603:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6747.408915] Lustre: server umount lustre-MDT0001 complete [ 6754.783197] Lustre: 332233:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761322757/real 1761322757] req@000000000792d41e x1846880702060224/t0(0) o39->lustre-MDT0001-lwp-OST2710@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761322763 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 6754.805700] Lustre: server umount lustre-OST2710 complete [ 6762.463164] Lustre: 332435:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761322765/real 1761322765] req@0000000039355ba9 x1846880702060672/t0(0) o39->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761322771 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 6762.542640] Lustre: server umount lustre-OST03e8 complete [ 6768.913721] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_hostid [ 6771.884763] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 6775.105588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6777.294952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6779.326296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6781.445756] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6783.757981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6783.783273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6783.889600] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6783.900302] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 6783.937796] Lustre: lustre-MDT0000: new disk, initializing [ 6783.961577] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6783.969040] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6785.404663] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 6789.628043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6789.652470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6789.691175] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6789.788832] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6791.431683] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 6798.477475] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6804.662898] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6811.627139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6811.712542] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6811.722094] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 6811.991567] Lustre: lustre-OST0000: new disk, initializing [ 6811.994886] Lustre: Skipped 1 previous similar message [ 6812.003595] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 6812.015165] Lustre: Skipped 2 previous similar messages [ 6812.092656] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6812.095512] Lustre: Skipped 1 previous similar message [ 6816.527175] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 6818.940745] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6818.949395] Lustre: Skipped 1 previous similar message [ 6818.952627] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6826.190340] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6831.351481] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 6831.679237] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6839.288470] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 6839.733590] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6844.895939] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6844.909642] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6844.911602] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6844.982249] LustreError: Skipped 1 previous similar message [ 6848.586251] Lustre: server umount lustre-OST0000 complete [ 6855.136445] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6855.138533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6855.140796] Lustre: Skipped 2 previous similar messages [ 6855.150268] Lustre: Skipped 2 previous similar messages [ 6859.674728] Lustre: server umount lustre-MDT0000 complete [ 6860.270269] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6860.290771] LustreError: Skipped 1 previous similar message [ 6861.283456] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6861.304894] LustreError: Skipped 1 previous similar message [ 6863.429733] LustreError: 337425:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761322872 with bad export cookie 575677082180778275 [ 6863.435210] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6863.445126] LustreError: 337425:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6870.412316] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 12:21:17 (1761322877) [ 6871.385997] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 6872.627656] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 12:21:19 (1761322879) [ 6874.790415] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 6880.060887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6882.404442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6884.516595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6886.815614] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6887.594893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6891.494515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6891.647375] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6891.661368] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6891.753210] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6894.585112] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 6901.003898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6901.080635] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6901.103183] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6903.793866] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 6908.498482] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6912.162953] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6916.492251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6916.534749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6916.642984] Lustre: Denying initial registration attempt from nid 192.168.201.151@tcp, specified as failover [ 6916.647616] LustreError: 160-7: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 6916.656816] LustreError: 343356:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -99 [ 6916.659885] LustreError: 343356:0:(obd_mount_server.c:1644:server_put_super()) no obd lustre-OST0000 [ 6916.662912] LustreError: 343356:0:(obd_mount_server.c:131:server_deregister_mount()) lustre-OST0000 not registered [ 6916.736116] Lustre: server umount lustre-OST0000 complete [ 6916.738150] Lustre: Skipped 1 previous similar message [ 6916.740046] LustreError: 343356:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 6921.185730] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6921.194434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6921.198971] Lustre: Skipped 1 previous similar message [ 6926.305733] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6926.311887] LustreError: Skipped 1 previous similar message [ 6928.500246] LustreError: 343628:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761322937 with bad export cookie 575677082180779108 [ 6928.507173] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6928.508614] LustreError: 343628:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6936.466288] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 6938.399286] Key type lgssc unregistered [ 6938.596844] LNet: 344431:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6938.603581] LNet: Removed LNI 192.168.201.151@tcp [ 6939.074714] Key type .llcrypt unregistered [ 6939.076487] Key type ._llcrypt unregistered [ 6946.846059] alg: No test for adler32 (adler32-zlib) [ 6947.597432] Key type ._llcrypt registered [ 6947.598843] Key type .llcrypt registered [ 6947.693926] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 6948.203136] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6948.573736] Lustre: Lustre: Build Version: 2.15.7_10_g040485f [ 6948.787033] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 6948.790981] LNet: Accept secure, port 988 [ 6950.431810] Key type lgssc registered [ 6951.155920] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6956.150112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6958.614713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6961.201164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6963.669548] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6964.373411] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6972.665846] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 6978.163621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6979.343459] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6979.359213] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6979.457085] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6981.567632] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 6986.964717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6987.075736] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6987.110260] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6987.401906] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6990.662672] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 6996.341901] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7000.281571] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7004.589597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7004.639701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7004.756679] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7004.876819] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7007.473826] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 7011.784609] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7017.445253] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7017.449597] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7017.453287] LustreError: Skipped 1 previous similar message [ 7017.454509] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7017.465973] Lustre: Skipped 1 previous similar message [ 7019.303363] Lustre: server umount lustre-OST0000 complete [ 7022.562188] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7022.562715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7022.569069] Lustre: Skipped 1 previous similar message [ 7022.573647] Lustre: Skipped 1 previous similar message [ 7027.679879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7027.684948] Lustre: Skipped 2 previous similar messages [ 7032.800407] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7032.804884] Lustre: Skipped 1 previous similar message [ 7036.383134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 7036.452421] Lustre: server umount lustre-MDT0000 complete [ 7037.920703] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7037.933088] LustreError: Skipped 1 previous similar message [ 7039.108215] LustreError: 349285:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761323048 with bad export cookie 2820636862516948636 [ 7039.117788] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7039.122432] LustreError: 349285:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7039.325994] Lustre: server umount lustre-MDT0001 complete [ 7046.065689] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 7047.762805] Key type lgssc unregistered [ 7047.953360] LNet: 351003:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7047.957185] LNet: Removed LNI 192.168.201.151@tcp [ 7048.495513] Key type .llcrypt unregistered [ 7048.497883] Key type ._llcrypt unregistered [ 7063.233344] alg: No test for adler32 (adler32-zlib) [ 7063.986427] Key type ._llcrypt registered [ 7063.988418] Key type .llcrypt registered [ 7064.064468] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 7064.594969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7064.837685] Lustre: Lustre: Build Version: 2.15.7_10_g040485f [ 7064.948243] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 7064.952227] LNet: Accept secure, port 988 [ 7066.591136] Key type lgssc registered [ 7067.199204] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7071.680130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7072.943179] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7072.982826] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7075.594653] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 7078.374834] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7080.749634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7080.992733] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7083.418111] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 7088.396322] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7092.003775] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7096.189585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7096.373962] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7098.823512] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 7103.082671] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7113.580359] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.51@tcp (stopping) [ 7113.700104] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7113.712459] Lustre: Skipped 2 previous similar messages [ 7115.061588] Lustre: server umount lustre-MDT0000 complete [ 7117.390950] LustreError: 352445:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761323126 with bad export cookie 10596985224264429413 [ 7117.398691] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7117.401656] LustreError: 352445:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7118.691266] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.201.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7118.713370] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.51@tcp (stopping) [ 7118.719226] Lustre: Skipped 3 previous similar messages [ 7118.820555] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7119.589506] Lustre: server umount lustre-MDT0001 complete [ 7123.690045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 7129.265647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7135.072905] Lustre: Evicted from MGS (at 192.168.201.151@tcp) after server handle changed from 0x93100dea8e582f65 to 0x93100dea8e584b3b [ 7135.080269] Lustre: MGC192.168.201.151@tcp: Connection restored to 192.168.201.151@tcp (at 0@lo) [ 7135.266823] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7135.277816] LustreError: Skipped 1 previous similar message [ 7135.328665] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7135.383118] Lustre: lustre-OST0000: deleting orphan objects from 0x0:166 to 0x0:193 [ 7137.900121] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 7139.362546] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7140.392813] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 7142.950979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7143.153460] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7145.409548] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 7150.073417] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7150.566826] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 7152.244639] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 1 sec [ 7155.840934] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7156.885810] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7160.801276] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7160.801395] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7160.806356] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7160.810245] Lustre: Skipped 1 previous similar message [ 7160.812893] Lustre: Skipped 1 previous similar message [ 7165.925092] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7165.929928] Lustre: Skipped 1 previous similar message [ 7171.040222] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7171.044583] Lustre: Skipped 1 previous similar message [ 7174.111503] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 7174.209419] Lustre: server umount lustre-OST0000 complete [ 7179.232503] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7179.236113] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7179.241817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7182.571330] Lustre: server umount lustre-MDT0000 complete [ 7184.887161] LustreError: 352443:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761323194 with bad export cookie 10596985224264436539 [ 7184.888806] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7184.900491] LustreError: 352443:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7192.961879] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 7194.653727] Key type lgssc unregistered [ 7194.860350] LNet: 358954:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7194.867655] LNet: Removed LNI 192.168.201.151@tcp [ 7195.316667] Key type .llcrypt unregistered [ 7195.318126] Key type ._llcrypt unregistered [ 7208.646770] alg: No test for adler32 (adler32-zlib) [ 7209.400396] Key type ._llcrypt registered [ 7209.401900] Key type .llcrypt registered [ 7209.459574] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 7210.049576] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7210.228727] Lustre: Lustre: Build Version: 2.15.7_10_g040485f [ 7210.345606] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 7210.351100] LNet: Accept secure, port 988 [ 7211.967181] Key type lgssc registered [ 7212.564451] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7216.801456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7218.039957] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7218.078972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7220.555902] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 7223.264673] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7226.234778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7226.429971] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7228.981363] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 7233.580057] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7237.204367] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7241.695967] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7241.699571] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7241.710019] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7242.720444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7242.720702] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7245.019422] Lustre: server umount lustre-MDT0000 complete [ 7246.970624] LustreError: 360395:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761323256 with bad export cookie 4820862856350863334 [ 7246.974713] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7246.977306] LustreError: 360395:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7247.134115] Lustre: server umount lustre-MDT0001 complete [ 7254.943613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7255.081658] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7255.100807] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 7255.217589] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7257.237597] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 7261.539237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7261.601227] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7261.624793] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 7261.778187] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7263.925736] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 7268.056723] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7272.094460] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7277.530506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7277.669470] LustreError: 13b-9: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 7277.677469] LustreError: 160-7: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 7277.682472] LustreError: 365107:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -2 [ 7277.686402] LustreError: 365107:0:(obd_mount_server.c:1644:server_put_super()) no obd lustre-OST0000 [ 7277.690463] LustreError: 365107:0:(obd_mount_server.c:131:server_deregister_mount()) lustre-OST0000 not registered [ 7277.770381] Lustre: server umount lustre-OST0000 complete [ 7277.772586] LustreError: 365107:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 7281.665248] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7281.790857] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7281.889397] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7284.050407] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 7288.195416] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7290.929055] Lustre: lustre-OST0000: deleting orphan objects from 0x0:166 to 0x0:225 [ 7292.584376] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7292.635162] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7292.642960] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 7292.676983] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7292.721944] Lustre: lustre-OST0001: new disk, initializing [ 7292.726034] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 7292.779313] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7295.035103] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 7298.941412] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7300.670032] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7300.683153] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7306.214166] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7306.220248] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7306.234666] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7306.239447] Lustre: Skipped 2 previous similar messages [ 7306.874302] Lustre: server umount lustre-OST0001 complete [ 7311.328761] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7311.330646] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7311.331301] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7311.331306] Lustre: Skipped 1 previous similar message [ 7311.332847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7311.338435] LustreError: Skipped 1 previous similar message [ 7311.344707] LustreError: Skipped 1 previous similar message [ 7315.274187] Lustre: server umount lustre-OST0000 complete [ 7317.984089] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7317.990333] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7317.994971] Lustre: Skipped 1 previous similar message [ 7317.997224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7317.999660] Lustre: Skipped 1 previous similar message [ 7323.407702] Lustre: server umount lustre-MDT0000 complete [ 7325.411185] LustreError: 365491:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761323334 with bad export cookie 4820862856350863887 [ 7325.413269] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7325.421409] LustreError: 365491:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7331.248498] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 7332.708050] Key type lgssc unregistered [ 7332.888273] LNet: 368472:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7332.892926] LNet: Removed LNI 192.168.201.151@tcp [ 7333.357472] Key type .llcrypt unregistered [ 7333.359157] Key type ._llcrypt unregistered [ 7352.902465] alg: No test for adler32 (adler32-zlib) [ 7353.656424] Key type ._llcrypt registered [ 7353.658674] Key type .llcrypt registered [ 7353.744726] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_hostid [ 7362.831225] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 7363.388850] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7363.605588] Lustre: Lustre: Build Version: 2.15.7_10_g040485f [ 7363.741972] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 7363.747042] LNet: Accept secure, port 988 [ 7365.383168] Key type lgssc registered [ 7365.954905] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7370.945107] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7374.694920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7378.203208] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7381.808586] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7385.699675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7385.753949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7386.886941] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 7386.910213] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 7386.958595] Lustre: lustre-MDT0000: new disk, initializing [ 7387.006979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7387.019169] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7389.155594] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 7394.951281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7394.995479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7395.045074] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 7395.068258] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 7395.071519] Lustre: Skipped 1 previous similar message [ 7395.118829] Lustre: lustre-MDT0001: new disk, initializing [ 7395.169130] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7395.205362] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7395.211509] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7397.321669] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 7403.142828] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7406.959546] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7411.117627] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7411.153955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7411.158109] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 7411.302699] Lustre: lustre-OST0000: new disk, initializing [ 7411.307299] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 7411.355148] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7413.779516] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 7416.348494] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7416.353814] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7419.250917] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7421.650960] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 7421.773785] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7424.185173] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 7424.310866] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7426.529429] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7426.530710] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7426.532525] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7426.539292] LustreError: Skipped 1 previous similar message [ 7426.544646] Lustre: Skipped 1 previous similar message [ 7431.556221] Lustre: server umount lustre-OST0000 complete [ 7436.256377] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7436.265272] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7436.281457] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7436.285605] Lustre: Skipped 1 previous similar message [ 7441.889164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7441.897078] Lustre: Skipped 2 previous similar messages [ 7447.009604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7447.015635] Lustre: Skipped 1 previous similar message [ 7448.031220] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 7448.116839] Lustre: server umount lustre-MDT0000 complete [ 7450.267645] LustreError: 373430:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761323459 with bad export cookie 557134905955513651 [ 7450.270155] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7450.272707] LustreError: 373430:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7450.429539] Lustre: server umount lustre-MDT0001 complete [ 7455.662844] Lustre: DEBUG MARKER: == conf-sanity test 60: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 12:31:02 (1761323462) [ 7459.182050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7462.460363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7474.416889] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_hostid [ 7478.397041] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 7483.410801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7486.896508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7490.247407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7493.508455] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7497.426212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7497.471313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7497.619363] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 7497.635444] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 7497.682568] Lustre: lustre-MDT0000: new disk, initializing [ 7497.738453] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7497.753575] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7500.091479] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 7506.423960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7506.473235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7506.520256] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 7506.547406] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 7506.550778] Lustre: Skipped 1 previous similar message [ 7506.597862] Lustre: lustre-MDT0001: new disk, initializing [ 7506.650336] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7506.671972] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7506.676905] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7509.171472] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 7514.850351] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7517.990261] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7521.474273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7521.509499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7521.513536] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 7521.648475] Lustre: lustre-OST0000: new disk, initializing [ 7521.651764] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 7521.717131] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7524.363627] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 7526.947325] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7526.952715] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7530.012227] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7532.238136] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 7532.345669] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7534.427074] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 7534.529924] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7537.123063] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7537.123906] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7537.127699] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7537.132315] Lustre: Skipped 2 previous similar messages [ 7541.636721] Lustre: server umount lustre-OST0000 complete [ 7547.361447] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7547.364907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7547.367770] Lustre: Skipped 1 previous similar message [ 7547.370530] Lustre: Skipped 1 previous similar message [ 7549.586221] Lustre: server umount lustre-MDT0000 complete [ 7551.402254] LustreError: 380331:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761323560 with bad export cookie 557134905955514498 [ 7551.404356] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7551.410237] LustreError: 380331:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7556.176391] Lustre: DEBUG MARKER: == conf-sanity test 61: large xattr ====================== 12:32:43 (1761323563) [ 7560.402881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7560.605624] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7560.644564] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7562.683589] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 7565.793698] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7566.629168] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7568.695221] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 7572.512333] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7575.421536] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7578.702666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7581.029554] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 7584.762202] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7599.986886] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61.conf-sanity [ 7601.080747] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61.conf-sanity [ 7602.245471] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61.conf-sanity [ 7603.260841] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61.conf-sanity after remounting MDS [ 7604.285933] Lustre: Failing over lustre-MDT0000 [ 7604.421897] Lustre: server umount lustre-MDT0000 complete [ 7604.423524] Lustre: Skipped 1 previous similar message [ 7605.617475] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.201.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7606.759549] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7606.768352] Lustre: Skipped 1 previous similar message [ 7610.717176] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.201.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7610.725426] LustreError: Skipped 3 previous similar messages [ 7612.895341] Lustre: 370803:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761323615/real 1761323615] req@0000000050827bc7 x1846881416162368/t0(0) o400->MGC192.168.201.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761323622 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 7612.908263] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7617.883117] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.201.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7617.891623] LustreError: Skipped 7 previous similar messages [ 7619.851846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7629.147676] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.201.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7629.158514] LustreError: Skipped 11 previous similar messages [ 7630.317035] Lustre: Evicted from MGS (at 192.168.201.151@tcp) after server handle changed from 0x7bb573ec172fbd1 to 0x7bb573ec172ffac [ 7630.327439] Lustre: MGC192.168.201.151@tcp: Connection restored to 192.168.201.151@tcp (at 0@lo) [ 7630.441335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7630.445792] Lustre: Skipped 2 previous similar messages [ 7630.473167] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7632.440110] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 7635.292341] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7635.944713] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 7635.967723] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 7635.990413] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 7639.066850] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7639.939214] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7642.393832] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61.conf-sanity [ 7643.424488] Lustre: Failing over lustre-MDT0000 [ 7643.594187] Lustre: server umount lustre-MDT0000 complete [ 7645.546550] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.201.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7645.554108] LustreError: Skipped 7 previous similar messages [ 7646.177647] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7646.178265] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7646.196280] Lustre: Skipped 3 previous similar messages [ 7653.343179] Lustre: 370804:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761323655/real 1761323655] req@000000004fb5d52f x1846881416171712/t0(0) o400->MGC192.168.201.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761323662 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 7653.357603] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7658.577888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7659.492346] Lustre: Evicted from MGS (at 192.168.201.151@tcp) after server handle changed from 0x7bb573ec172ffac to 0x7bb573ec17302b5 [ 7659.505164] Lustre: MGC192.168.201.151@tcp: Connection restored to (at 0@lo) [ 7659.507656] Lustre: Skipped 2 previous similar messages [ 7659.655429] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7661.923612] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 7662.938831] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7665.128055] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 7665.149856] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 7665.183927] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:97 [ 7668.202845] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7669.003967] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7675.362325] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7675.363086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7675.367962] Lustre: Skipped 2 previous similar messages [ 7675.371105] Lustre: Skipped 2 previous similar messages [ 7675.376238] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7678.043152] Lustre: server umount lustre-MDT0000 complete [ 7679.865734] LustreError: 384966:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761323689 with bad export cookie 557134905955517109 [ 7679.867057] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7679.871160] LustreError: 384966:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7688.159414] Lustre: 390792:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761323691/real 1761323691] req@00000000991189a7 x1846881416181184/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761323697 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 7696.409213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7696.648721] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7696.657052] LustreError: Skipped 18 previous similar messages [ 7696.687734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7696.691761] Lustre: Skipped 1 previous similar message [ 7698.748244] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 7703.291722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7705.543604] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 7708.939379] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7711.599939] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7714.718848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7717.126642] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 7721.738552] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7727.205541] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:129 [ 7732.704811] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7732.706478] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7732.710511] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7732.714672] Lustre: Skipped 1 previous similar message [ 7732.717581] Lustre: Skipped 1 previous similar message [ 7743.575209] Lustre: server umount lustre-MDT0000 complete [ 7743.584172] Lustre: Skipped 3 previous similar messages [ 7745.296199] LustreError: 391598:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761323754 with bad export cookie 557134905955517816 [ 7745.299034] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7745.300412] LustreError: 391598:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7751.288492] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 7752.551423] Key type lgssc unregistered [ 7752.686522] LNet: 395268:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7752.691737] LNet: Removed LNI 192.168.201.151@tcp [ 7753.050061] Key type .llcrypt unregistered [ 7753.051561] Key type ._llcrypt unregistered [ 7764.731532] alg: No test for adler32 (adler32-zlib) [ 7765.485451] Key type ._llcrypt registered [ 7765.486952] Key type .llcrypt registered [ 7765.531411] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 7766.019067] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7766.183202] Lustre: Lustre: Build Version: 2.15.7_10_g040485f [ 7766.278257] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 7766.283463] LNet: Accept secure, port 988 [ 7767.879271] Key type lgssc registered [ 7768.358387] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7772.234639] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7772.238368] LustreError: 396752:0:(osd_handler.c:8211:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 7772.241461] LustreError: 396752:0:(obd_config.c:774:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 7772.243573] LustreError: 396752:0:(obd_mount.c:200:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 7772.246487] LustreError: 396752:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 7772.250393] LustreError: 396752:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7775.919206] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7775.922776] LustreError: 397149:0:(osd_handler.c:8211:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 7775.928394] LustreError: 397149:0:(obd_config.c:774:class_setup()) setup lustre-OST0000-osd failed (-22) [ 7775.933235] LustreError: 397149:0:(obd_mount.c:200:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 7775.936722] LustreError: 397149:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 7775.940971] LustreError: 397149:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7782.888452] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 7784.075442] Key type lgssc unregistered [ 7784.212713] LNet: 398105:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7784.216512] LNet: Removed LNI 192.168.201.151@tcp [ 7784.576082] Key type .llcrypt unregistered [ 7784.577637] Key type ._llcrypt unregistered [ 7791.964050] alg: No test for adler32 (adler32-zlib) [ 7792.716406] Key type ._llcrypt registered [ 7792.717960] Key type .llcrypt registered [ 7792.759073] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_hostid [ 7799.496851] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 7799.850480] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7800.132700] Lustre: Lustre: Build Version: 2.15.7_10_g040485f [ 7800.230989] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 7800.233743] LNet: Accept secure, port 988 [ 7801.831194] Key type lgssc registered [ 7802.292548] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7806.001953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7808.801192] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7811.436361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7814.124215] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7816.944281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7816.973249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7818.086861] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 7818.106225] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 7818.146544] Lustre: lustre-MDT0000: new disk, initializing [ 7818.189204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7818.199243] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7820.061754] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 7825.006115] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7825.041792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7825.086303] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 7825.108352] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 7825.110616] Lustre: Skipped 1 previous similar message [ 7825.150475] Lustre: lustre-MDT0001: new disk, initializing [ 7825.199241] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7825.213871] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7825.218057] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7827.072733] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 7831.871917] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7834.313030] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7837.112781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7837.141242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7837.145304] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 7837.253710] Lustre: lustre-OST0000: new disk, initializing [ 7837.256839] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 7837.293389] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7839.312294] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 7844.165488] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7844.363021] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7844.371774] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7846.224540] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 7846.346900] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7848.289731] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 7848.398330] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7849.442466] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7849.442715] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7849.445983] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7849.448424] Lustre: Skipped 1 previous similar message [ 7851.374669] Lustre: server umount lustre-OST0000 complete [ 7854.562294] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7854.565073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7854.571666] Lustre: Skipped 1 previous similar message [ 7854.576383] Lustre: Skipped 1 previous similar message [ 7859.282971] Lustre: server umount lustre-MDT0000 complete [ 7859.681334] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7859.691505] LustreError: Skipped 1 previous similar message [ 7861.207934] LustreError: 401370:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761323870 with bad export cookie 67536625242593757 [ 7861.209861] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7861.219622] LustreError: 401370:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7861.371788] Lustre: server umount lustre-MDT0001 complete [ 7865.256917] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 7866.093840] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 12:37:53 (1761323873) [ 7869.027416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7869.241046] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7869.270215] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7871.244620] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 7874.529603] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7874.863443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7875.012795] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7876.928510] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 7880.205726] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7882.997702] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7886.066249] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7886.236886] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7888.055762] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 7891.239261] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7894.094953] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7894.129256] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7894.133224] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 7894.185477] Lustre: lustre-OST0001: new disk, initializing [ 7894.188302] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 7895.881666] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7895.885746] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7896.049906] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 7900.369334] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7904.227154] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7904.228065] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7904.232025] Lustre: Skipped 1 previous similar message [ 7904.237068] Lustre: Skipped 1 previous similar message [ 7904.427532] Lustre: server umount lustre-OST0001 complete [ 7905.210264] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 192.168.201.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7909.350345] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7909.355859] LustreError: Skipped 1 previous similar message [ 7919.585909] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7919.586397] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7919.592872] LustreError: Skipped 3 previous similar messages [ 7919.598960] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7919.599490] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7919.606454] LustreError: Skipped 1 previous similar message [ 7924.705194] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7924.707707] Lustre: Skipped 2 previous similar messages [ 7931.359529] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 7931.434995] Lustre: server umount lustre-OST0000 complete [ 7934.946305] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7934.947706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7934.951859] Lustre: Skipped 2 previous similar messages [ 7934.953914] Lustre: Skipped 2 previous similar messages [ 7940.066139] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7940.073277] LustreError: Skipped 5 previous similar messages [ 7941.105702] LustreError: 408874:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761323950 with bad export cookie 67536625242594597 [ 7941.109144] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7941.114082] LustreError: 408874:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7947.353598] Lustre: server umount lustre-MDT0001 complete [ 7947.356446] Lustre: Skipped 1 previous similar message [ 7953.048992] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 7954.246160] Key type lgssc unregistered [ 7954.373887] LNet: 410878:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7954.378206] LNet: Removed LNI 192.168.201.151@tcp [ 7954.728639] Key type .llcrypt unregistered [ 7954.730433] Key type ._llcrypt unregistered [ 7969.083946] alg: No test for adler32 (adler32-zlib) [ 7969.835426] Key type ._llcrypt registered [ 7969.836954] Key type .llcrypt registered [ 7969.883493] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_hostid [ 7977.333523] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 7977.780926] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7977.960058] Lustre: Lustre: Build Version: 2.15.7_10_g040485f [ 7978.056480] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 7978.060409] LNet: Accept secure, port 988 [ 7979.663148] Key type lgssc registered [ 7980.106345] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7984.055634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7987.597568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7990.350713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7993.138802] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7996.233428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7996.272076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7997.403918] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 7997.420273] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 7997.468804] Lustre: lustre-MDT0000: new disk, initializing [ 7997.525149] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7997.533953] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7999.418866] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 8004.643268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8004.682609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8004.727662] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 8004.752384] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 8004.755412] Lustre: Skipped 1 previous similar message [ 8004.796879] Lustre: lustre-MDT0001: new disk, initializing [ 8004.840906] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8004.865570] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8004.869940] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8006.797154] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 8011.526395] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8014.034220] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8017.006481] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8017.041071] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8017.044258] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 8017.157783] Lustre: lustre-OST0000: new disk, initializing [ 8017.162452] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 8017.199268] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8018.979190] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 8023.345415] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8025.276836] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 8026.630600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8026.635881] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8027.425978] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 8029.340562] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 8029.432143] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8031.712159] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8031.716349] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8031.719624] LustreError: Skipped 1 previous similar message [ 8031.720478] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8031.733322] Lustre: Skipped 1 previous similar message [ 8032.426499] Lustre: server umount lustre-OST0000 complete [ 8035.808187] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8035.812775] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8035.818409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8035.821183] Lustre: Skipped 1 previous similar message [ 8036.832312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8036.832448] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8040.199687] Lustre: server umount lustre-MDT0000 complete [ 8041.772149] LustreError: 414959:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761324050 with bad export cookie 2549190771598279426 [ 8041.774684] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8041.777758] LustreError: 414959:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8041.903955] Lustre: server umount lustre-MDT0001 complete [ 8045.834355] Lustre: DEBUG MARKER: == conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================================================== 12:40:53 (1761324053) [ 8049.327911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8052.865765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8053.070188] Lustre: lustre-MDT0000: new disk, initializing [ 8053.086041] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8053.119663] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8054.911921] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 8058.336347] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8058.747415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8058.926914] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8060.895316] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 8064.501528] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8067.161465] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8068.065584] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8068.066643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8068.074560] Lustre: Skipped 1 previous similar message [ 8068.078697] Lustre: Skipped 2 previous similar messages [ 8073.184302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8073.186853] Lustre: Skipped 1 previous similar message [ 8082.399125] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 8082.438727] Lustre: server umount lustre-MDT0000 complete [ 8083.879214] LustreError: 420104:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761324093 with bad export cookie 2549190771598280266 [ 8083.881659] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8083.884040] LustreError: 420104:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8087.444786] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 12:41:35 (1761324095) [ 8090.136522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8090.347395] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8090.382682] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8092.250774] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 8095.713067] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8096.007441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8097.875767] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 8101.009294] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8103.256565] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8105.908515] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8107.851194] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 8110.846496] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8124.350263] Lustre: Permanently deactivating lustre-OST0000 [ 8124.356827] Lustre: Setting parameter lustre-OST0000-osc.osc.active in log lustre-client [ 8134.131992] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8135.868155] LustreError: 425980:0:(mgs_llog.c:1551:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002020 [ 8135.872938] LustreError: 425980:0:(mgs_llog.c:1605:mgs_replace_nids()) Only MGS is allowed to be started [ 8135.876040] LustreError: 425980:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -115 [ 8136.768368] LustreError: 426028:0:(mgs_llog.c:1605:mgs_replace_nids()) Only MGS is allowed to be started [ 8136.771348] LustreError: 426028:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -115 [ 8143.709241] Lustre: server umount lustre-OST0000 complete [ 8143.710807] Lustre: Skipped 1 previous similar message [ 8144.829134] LustreError: 426277:0:(mgs_llog.c:1605:mgs_replace_nids()) Only MGS is allowed to be started [ 8144.831717] LustreError: 426277:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -115 [ 8146.912226] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8146.912647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8146.916733] Lustre: Skipped 1 previous similar message [ 8146.918777] Lustre: Skipped 4 previous similar messages [ 8151.007126] Lustre: 426374:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761324154/real 1761324154] req@0000000055bf6d6a x1846882061030592/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761324160 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 8152.032213] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8152.040491] LustreError: Skipped 1 previous similar message [ 8152.785373] LustreError: 426375:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761324161 with bad export cookie 2549190771598280805 [ 8152.786900] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8152.790186] LustreError: 426375:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8159.199155] Lustre: 426576:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761324162/real 1761324162] req@000000000341f34e x1846882061031552/t0(0) o9->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761324168 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 8159.365328] Lustre: server umount lustre-MDT0001 complete [ 8159.367736] Lustre: Skipped 1 previous similar message [ 8162.998690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8164.792551] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 8165.886610] LustreError: 427485:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -6 [ 8168.486988] Lustre: 427830:0:(mgs_llog.c:1311:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.151@tcp [ 8175.583177] Lustre: 427929:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761324178/real 1761324178] req@0000000053b22e5d x1846882061032000/t0(0) o251->MGC192.168.201.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761324184 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 8175.597445] Lustre: server umount MGS complete [ 8180.553311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8180.792701] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8180.801164] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8180.803597] Lustre: Skipped 1 previous similar message [ 8180.826186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8180.828606] Lustre: Skipped 2 previous similar messages [ 8182.617453] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 8186.240368] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8186.384884] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8188.260719] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 8191.317165] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8193.527973] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8194.292761] Lustre: Permanently reactivating lustre-OST0000 [ 8194.299567] Lustre: Modifying parameter lustre-OST0000-osc.osc.active in log lustre-client [ 8194.303025] Lustre: Skipped 2 previous similar messages [ 8214.668361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8214.797892] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8214.800102] Lustre: Skipped 1 previous similar message [ 8216.524412] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 8219.517079] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8236.001260] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 8236.004402] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8236.009717] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8238.302123] Lustre: server umount lustre-OST0000 complete [ 8246.752945] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8246.759156] LustreError: Skipped 2 previous similar messages [ 8247.210255] LustreError: 428538:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761324256 with bad export cookie 2549190771598281750 [ 8247.212635] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8247.214865] LustreError: 428538:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8252.517319] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 8253.517624] Key type lgssc unregistered [ 8253.641287] LNet: 433010:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8253.643457] LNet: Removed LNI 192.168.201.151@tcp [ 8253.934476] Key type .llcrypt unregistered [ 8253.935889] Key type ._llcrypt unregistered [ 8260.841360] alg: No test for adler32 (adler32-zlib) [ 8261.593530] Key type ._llcrypt registered [ 8261.595488] Key type .llcrypt registered [ 8261.642498] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_hostid [ 8267.835368] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 8268.206375] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8268.430112] Lustre: Lustre: Build Version: 2.15.7_10_g040485f [ 8268.528366] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 8268.531120] LNet: Accept secure, port 988 [ 8270.127114] Key type lgssc registered [ 8270.522079] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8274.093737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8276.578521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8278.798606] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8281.009524] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8283.734231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8283.771435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8284.878040] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 8284.891731] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 8284.933524] Lustre: lustre-MDT0000: new disk, initializing [ 8284.968228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8284.976841] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8286.736549] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 8291.224825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8291.248019] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8291.281483] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 8291.298997] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 8291.301466] Lustre: Skipped 1 previous similar message [ 8291.340700] Lustre: lustre-MDT0001: new disk, initializing [ 8291.381574] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8291.395996] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8291.399964] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8292.829258] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 8296.779924] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8298.677615] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8300.970734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8300.995508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8300.998408] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 8301.096168] Lustre: lustre-OST0000: new disk, initializing [ 8301.098616] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 8301.128436] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8302.690793] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 8304.885500] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8304.889522] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8306.794196] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8308.382142] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 8308.462201] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8310.059934] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 8310.153882] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8315.360212] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8315.361715] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8315.363626] LustreError: Skipped 1 previous similar message [ 8315.370081] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8315.374407] Lustre: Skipped 1 previous similar message [ 8320.481388] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8320.483629] Lustre: Skipped 1 previous similar message [ 8325.087097] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 8325.142521] Lustre: server umount lustre-OST0000 complete [ 8327.135752] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8327.138167] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8327.141419] Lustre: Skipped 1 previous similar message [ 8327.143880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8330.720139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8330.720659] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8332.548988] Lustre: server umount lustre-MDT0000 complete [ 8333.936972] LustreError: 439960:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761324343 with bad export cookie 15072724665419249885 [ 8333.939499] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8333.940923] LustreError: 439960:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8334.055087] Lustre: server umount lustre-MDT0001 complete [ 8336.853950] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 8337.442076] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 12:45:45 (1761324345) [ 8339.745265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8339.890163] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8339.912242] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8341.391608] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 8344.585603] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8344.694912] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8346.237793] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 8349.095909] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8351.227758] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8353.618517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8353.739304] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8355.378756] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 8358.075134] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8358.797699] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 8360.316465] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 8364.001626] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8364.004654] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8364.009053] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8364.010511] Lustre: Skipped 2 previous similar messages [ 8367.836458] Lustre: server umount lustre-OST0000 complete [ 8370.656079] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8370.659792] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8370.663912] Lustre: Skipped 1 previous similar message [ 8374.240365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8374.243205] Lustre: Skipped 2 previous similar messages [ 8375.302074] Lustre: server umount lustre-MDT0000 complete [ 8376.519893] LustreError: 441665:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761324385 with bad export cookie 15072724665419250739 [ 8376.522255] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8376.524876] LustreError: 441665:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8381.260368] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 8382.291707] Key type lgssc unregistered [ 8382.404491] LNet: 444723:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8382.407468] LNet: Removed LNI 192.168.201.151@tcp [ 8382.763404] Key type .llcrypt unregistered [ 8382.764993] Key type ._llcrypt unregistered [ 8391.996056] alg: No test for adler32 (adler32-zlib) [ 8392.747614] Key type ._llcrypt registered [ 8392.749226] Key type .llcrypt registered [ 8392.794145] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 8392.891816] Key type .llcrypt unregistered [ 8392.893185] Key type ._llcrypt unregistered [ 8399.741402] alg: No test for adler32 (adler32-zlib) [ 8400.493422] Key type ._llcrypt registered [ 8400.494929] Key type .llcrypt registered [ 8400.536195] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 8400.885230] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8401.116988] Lustre: Lustre: Build Version: 2.15.7_10_g040485f [ 8401.198818] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 8401.200397] LNet: Accept secure, port 988 [ 8402.783200] Key type lgssc registered [ 8403.175868] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8406.018983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8407.174491] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8407.189423] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8408.608424] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 8411.491183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8411.594669] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8411.599155] LustreError: Skipped 1 previous similar message [ 8411.602115] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8412.640531] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8412.646297] LustreError: Skipped 1 previous similar message [ 8412.647769] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 8413.040659] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 8415.741914] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8417.762935] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8417.768115] LustreError: Skipped 1 previous similar message [ 8418.043598] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8418.171431] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8419.749732] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 8427.999767] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8428.002058] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8428.007167] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8428.513986] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8428.514155] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8428.517568] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8433.115772] Lustre: server umount lustre-OST0000 complete [ 8437.729630] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 8437.732911] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8437.738172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8437.740471] Lustre: Skipped 1 previous similar message [ 8440.769769] Lustre: server umount lustre-MDT0000 complete [ 8442.218280] LustreError: 448800:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761324451 with bad export cookie 1857572065893084482 [ 8442.220707] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8442.223248] LustreError: 448800:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8442.388811] Lustre: server umount lustre-MDT0001 complete [ 8447.561949] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 8448.636959] Key type lgssc unregistered [ 8448.757503] LNet: 450310:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8448.760428] LNet: Removed LNI 192.168.201.151@tcp [ 8449.087660] Key type .llcrypt unregistered [ 8449.089113] Key type ._llcrypt unregistered [ 8458.691462] alg: No test for adler32 (adler32-zlib) [ 8459.443371] Key type ._llcrypt registered [ 8459.445152] Key type .llcrypt registered [ 8459.489949] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 8459.882486] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8460.171286] Lustre: Lustre: Build Version: 2.15.7_10_g040485f [ 8460.261131] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 8460.263638] LNet: Accept secure, port 988 [ 8461.863142] Key type lgssc registered [ 8462.292326] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8465.309620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8482.783400] LustreError: 451754:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.201.151@tcp: failed processing log, type 1: rc = -5 [ 8513.503211] LustreError: 451754:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.201.151@tcp: failed processing log, type 4: rc = -110 [ 8543.199169] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8544.624959] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 8547.135110] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8549.385763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8565.782704] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8565.810274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8565.835695] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:97 [ 8567.359830] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 8570.344975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8570.467654] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8571.859425] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 8574.381939] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8576.508409] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8580.576037] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8580.579050] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8580.583258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8581.088515] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8581.090084] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8581.090265] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8581.099269] Lustre: Skipped 1 previous similar message [ 8585.066715] Lustre: server umount lustre-OST0000 complete [ 8589.793323] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 8589.796310] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8589.800636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8592.451321] Lustre: server umount lustre-MDT0000 complete [ 8593.792201] LustreError: 451762:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761324603 with bad export cookie 2902698286293593249 [ 8593.796224] LustreError: 451762:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8593.796352] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8593.905631] Lustre: server umount lustre-MDT0001 complete [ 8598.460359] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 8599.503663] Key type lgssc unregistered [ 8599.615286] LNet: 455448:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8599.617849] LNet: Removed LNI 192.168.201.151@tcp [ 8599.898628] Key type .llcrypt unregistered [ 8599.899701] Key type ._llcrypt unregistered [ 8609.066583] alg: No test for adler32 (adler32-zlib) [ 8609.818283] Key type ._llcrypt registered [ 8609.819167] Key type .llcrypt registered [ 8609.854755] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 8610.279790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8610.395134] Lustre: Lustre: Build Version: 2.15.7_10_g040485f [ 8610.466709] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 8610.468265] LNet: Accept secure, port 988 [ 8612.047107] Key type lgssc registered [ 8612.404915] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8615.240588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8616.391710] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8616.406894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8617.935893] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 8620.744051] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8620.871154] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8622.296250] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 8624.801809] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8626.721644] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8628.988387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8629.091417] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8630.433876] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 8632.167285] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:129 [ 8632.822709] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8636.257263] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8636.257914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8636.264223] Lustre: Skipped 2 previous similar messages [ 8636.266583] Lustre: Skipped 1 previous similar message [ 8639.329282] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.51@tcp (stopping) [ 8640.774921] Lustre: server umount lustre-MDT0000 complete [ 8641.504768] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8641.512617] LustreError: Skipped 2 previous similar messages [ 8641.604712] LustreError: 459553:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.51@tcp arrived at 1761324650 with bad export cookie 4666402823055225894 [ 8641.614250] LustreError: 459553:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8642.303962] LustreError: 459553:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761324651 with bad export cookie 4666402823055225019 [ 8642.307202] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8642.307965] LustreError: 459553:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8646.624154] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8646.624289] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8646.624298] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8646.624883] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8646.628781] LustreError: Skipped 1 previous similar message [ 8648.418487] Lustre: server umount lustre-OST0000 complete [ 8648.672467] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8648.678400] LustreError: Skipped 3 previous similar messages [ 8650.844987] Lustre: server umount lustre-MDT0001 complete [ 8655.532673] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 8656.529316] Key type lgssc unregistered [ 8656.657280] LNet: 460710:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8656.660236] LNet: Removed LNI 192.168.201.151@tcp [ 8656.953139] Key type .llcrypt unregistered [ 8656.954210] Key type ._llcrypt unregistered [ 8665.670725] alg: No test for adler32 (adler32-zlib) [ 8666.422322] Key type ._llcrypt registered [ 8666.423620] Key type .llcrypt registered [ 8666.455875] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 8666.868702] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8667.015771] Lustre: Lustre: Build Version: 2.15.7_10_g040485f [ 8667.094355] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 8667.095882] LNet: Accept secure, port 988 [ 8668.679095] Key type lgssc registered [ 8669.023251] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8671.643185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8672.774566] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8672.789669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8674.257063] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 8676.832568] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8677.315025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8677.447576] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8678.978245] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 8681.618308] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8683.597462] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8686.021592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8686.193246] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8687.207058] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:161 [ 8687.675782] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 8690.253298] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8692.707124] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 8692.707230] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8692.707971] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8692.718690] Lustre: Skipped 1 previous similar message [ 8696.667283] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.51@tcp (stopping) [ 8696.675699] Lustre: Skipped 1 previous similar message [ 8697.825093] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8697.827590] Lustre: Skipped 1 previous similar message [ 8698.172055] Lustre: server umount lustre-MDT0001 complete [ 8702.944367] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8702.944500] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8702.944564] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8702.958476] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8705.944443] Lustre: server umount lustre-OST0000 complete [ 8713.695240] Lustre: 465220:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761324716/real 1761324716] req@000000001d1b920b x1846882783472384/t0(0) o251->MGC192.168.201.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761324722 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 8713.783865] Lustre: server umount lustre-MDT0000 complete [ 8719.547781] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 8720.585970] Key type lgssc unregistered [ 8720.711390] LNet: 465975:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8720.713630] LNet: Removed LNI 192.168.201.151@tcp [ 8721.016784] Key type .llcrypt unregistered [ 8721.017759] Key type ._llcrypt unregistered [ 8730.180515] alg: No test for adler32 (adler32-zlib) [ 8730.932355] Key type ._llcrypt registered [ 8730.933744] Key type .llcrypt registered [ 8730.973811] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_hostid [ 8736.289721] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 8736.663506] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8736.886837] Lustre: Lustre: Build Version: 2.15.7_10_g040485f [ 8736.980880] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 8736.983653] LNet: Accept secure, port 988 [ 8738.567166] Key type lgssc registered [ 8738.924213] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8741.769321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8743.943085] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8746.019151] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8748.225708] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8751.178824] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 8754.437057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8756.765726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8756.787562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8757.869658] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 8757.882495] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 8757.924599] Lustre: lustre-MDT0000: new disk, initializing [ 8757.948413] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8757.955609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8759.300018] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 8762.770831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8764.878497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8764.901699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8764.905472] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 8764.992744] Lustre: lustre-OST0000: new disk, initializing [ 8764.994565] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 8764.997208] Lustre: Skipped 1 previous similar message [ 8765.018663] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8766.409932] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 8770.226084] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8778.686079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8781.013271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8781.040074] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8781.085312] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 8781.111717] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 8781.152069] Lustre: lustre-MDT0001: new disk, initializing [ 8781.175353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 8781.182795] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 8781.198856] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8782.728173] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 8786.655519] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 40 [ 8786.733494] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 8788.422743] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 40 [ 8788.509629] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 8792.545171] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8792.545296] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 8792.550689] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8792.555598] Lustre: Skipped 1 previous similar message [ 8793.955634] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.51@tcp (stopping) [ 8796.353307] Lustre: server umount lustre-MDT0001 complete [ 8797.665763] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8797.670088] LustreError: Skipped 1 previous similar message [ 8802.784718] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8802.789176] LustreError: Skipped 1 previous similar message [ 8807.904529] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8807.909769] LustreError: Skipped 1 previous similar message [ 8811.999728] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8812.002965] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8812.006598] Lustre: Skipped 1 previous similar message [ 8812.008162] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8813.024530] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8818.144153] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8818.144522] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8818.145741] Lustre: Skipped 1 previous similar message [ 8823.263126] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 8823.264876] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8823.319261] Lustre: server umount lustre-OST0000 complete [ 8837.087158] Lustre: 474376:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761324840/real 1761324840] req@00000000498bb9c0 x1846882856878080/t0(0) o251->MGC192.168.201.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761324846 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 8837.168596] Lustre: server umount lustre-MDT0000 complete [ 8843.196667] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 8844.256476] Key type lgssc unregistered [ 8844.375398] LNet: 475131:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8844.379864] LNet: Removed LNI 192.168.201.151@tcp [ 8844.685916] Key type .llcrypt unregistered [ 8844.687823] Key type ._llcrypt unregistered [ 8859.572264] alg: No test for adler32 (adler32-zlib) [ 8860.323393] Key type ._llcrypt registered [ 8860.324930] Key type .llcrypt registered [ 8860.360620] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 8860.456284] Key type .llcrypt unregistered [ 8860.457758] Key type ._llcrypt unregistered [ 8866.685515] alg: No test for adler32 (adler32-zlib) [ 8867.437250] Key type ._llcrypt registered [ 8867.438052] Key type .llcrypt registered [ 8867.471114] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 8867.899344] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8868.026426] Lustre: Lustre: Build Version: 2.15.7_10_g040485f [ 8868.097248] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 8868.098935] LNet: Accept secure, port 988 [ 8869.671049] Key type lgssc registered [ 8870.013594] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8872.433193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8874.550190] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8876.612821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8878.809141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8878.838434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8879.940996] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 8879.951830] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 8879.993489] Lustre: lustre-MDT0000: new disk, initializing [ 8880.015688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8880.021521] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8881.385644] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 8885.615213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8885.646661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8885.685845] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 8885.705636] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 8885.709241] Lustre: Skipped 1 previous similar message [ 8885.752889] Lustre: lustre-MDT0001: new disk, initializing [ 8885.791382] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8885.807995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8885.813226] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8887.330230] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 8891.023497] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8892.899245] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8895.279758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8895.308584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8895.311474] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 8895.404684] Lustre: lustre-OST0000: new disk, initializing [ 8895.407354] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 8895.438755] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8897.006447] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 8900.944559] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8902.645452] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8902.649575] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8911.327833] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8911.330261] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8911.334069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8912.864637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8912.865702] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8912.871132] Lustre: Skipped 1 previous similar message [ 8914.754925] Lustre: server umount lustre-MDT0000 complete [ 8916.060888] LustreError: 478614:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761324925 with bad export cookie 17803891379931109831 [ 8916.063035] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8916.063666] LustreError: 478614:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8916.190395] Lustre: server umount lustre-MDT0001 complete [ 8923.615106] Lustre: 481877:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761324926/real 1761324926] req@0000000023d28039 x1846882994239808/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761324932 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 8923.687280] Lustre: server umount lustre-OST0000 complete [ 8927.022855] Lustre: DEBUG MARKER: == conf-sanity test 73: failnode to update from mountdata properly ========================================================== 12:55:34 (1761324934) [ 8927.370899] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8929.545618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8929.677222] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8929.695936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8930.992323] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 8933.719395] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8933.823494] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8935.182484] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 8937.672865] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8939.554720] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8941.730618] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8941.759849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8941.835583] Lustre: Found index 0 for lustre-OST0000, updating log [ 8941.898933] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8943.359732] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 8944.995955] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5 to 0x0:33 [ 8945.616963] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8949.088625] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8949.089242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8949.093509] Lustre: Skipped 3 previous similar messages [ 8952.787447] Lustre: server umount lustre-OST0000 complete [ 8954.336087] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8954.336149] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8954.339622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8954.341301] Lustre: Skipped 1 previous similar message [ 8959.455912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8959.458768] Lustre: Skipped 2 previous similar messages [ 8966.493599] Lustre: server umount lustre-MDT0000 complete [ 8967.719955] LustreError: 482681:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761324976 with bad export cookie 17803891379931111154 [ 8967.722626] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8967.726226] LustreError: 482681:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8970.532853] Lustre: DEBUG MARKER: == conf-sanity test 75: The order of --index should be irrelevant ========================================================== 12:56:18 (1761324978) [ 8972.383627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8973.984295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8975.739878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8977.333386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8982.810948] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_hostid [ 8985.369978] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 8988.834757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8991.068264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8993.098057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8995.206198] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8997.544102] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8997.569338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8997.652768] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 8997.663857] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 8997.702611] Lustre: lustre-MDT0000: new disk, initializing [ 8997.730672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8997.738814] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8999.254990] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 9003.573891] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9003.600026] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9003.634332] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 9003.653541] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 9003.655919] Lustre: Skipped 1 previous similar message [ 9003.695734] Lustre: lustre-MDT0001: new disk, initializing [ 9003.740161] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9003.743292] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9005.274499] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 9009.140952] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9010.955841] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9013.272713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9013.301350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9013.304994] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 9013.395519] Lustre: lustre-OST0000: new disk, initializing [ 9013.398351] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 9014.915943] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all 8 [ 9018.790149] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9020.398303] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 9022.966333] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9022.970562] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9023.553141] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 9025.025194] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 9025.111783] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9028.064370] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9028.065731] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9028.067221] LustreError: Skipped 1 previous similar message [ 9028.071390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9028.073909] Lustre: Skipped 5 previous similar messages [ 9031.893898] Lustre: server umount lustre-OST0000 complete [ 9031.895361] Lustre: Skipped 1 previous similar message [ 9034.719834] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9038.304338] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9038.308901] Lustre: Skipped 2 previous similar messages [ 9040.508961] LustreError: 489706:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761325049 with bad export cookie 17803891379931112197 [ 9040.510760] LustreError: 166-1: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9040.513935] LustreError: 489706:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9048.194276] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_hostid [ 9051.170214] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 9054.798312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9057.330489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9059.341604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9061.706823] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9062.245947] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 8867 sec ========== 12:57:50 (1761325070) [ 9078.448428] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 9079.411872] Key type lgssc unregistered [ 9079.518345] LNet: 498458:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9079.521055] LNet: Removed LNI 192.168.201.151@tcp [ 9079.785855] Key type .llcrypt unregistered [ 9079.786967] Key type ._llcrypt unregistered