[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 484025908 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.003199] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005020] kvm-guest: setup PV IPIs [ 0.008325] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.011005] pid_max: default: 32768 minimum: 301 [ 0.012150] LSM: Security Framework initializing [ 0.013081] Yama: becoming mindful. [ 0.014052] SELinux: Initializing. [ 0.015089] *** VALIDATE selinux *** [ 0.024524] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029128] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030167] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032112] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033122] *** VALIDATE tmpfs *** [ 0.035215] *** VALIDATE proc *** [ 0.037185] *** VALIDATE cgroup *** [ 0.038015] *** VALIDATE cgroup2 *** [ 0.039291] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.040163] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.041010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.042032] Spectre V2 : User space: Vulnerable [ 0.043012] Speculative Store Bypass: Vulnerable [ 0.045891] debug: unmapping init [mem 0xffffffffa8059000-0xffffffffa8060fff] [ 0.048000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048732] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049033] ... version: 2 [ 0.050018] ... bit width: 48 [ 0.051012] ... generic registers: 4 [ 0.052012] ... value mask: 0000ffffffffffff [ 0.053017] ... max period: 00007fffffffffff [ 0.054018] ... fixed-purpose events: 3 [ 0.055016] ... event mask: 000000070000000f [ 0.057287] rcu: Hierarchical SRCU implementation. [ 0.059498] smp: Bringing up secondary CPUs ... [ 0.060672] x86: Booting SMP configuration: [ 0.061031] .... node #0, CPUs: #1 #2 #3 [ 0.065207] smp: Brought up 1 node, 4 CPUs [ 0.067018] smpboot: Max logical packages: 1 [ 0.068027] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.258039] node 0 deferred pages initialised in 187ms [ 0.262193] devtmpfs: initialized [ 0.263257] x86/mm: Memory block size: 128MB [ 0.265499] gcov: version magic: 0x41383552 [ 0.267569] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.270157] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.272452] pinctrl core: initialized pinctrl subsystem [ 0.274257] [ 0.274821] ************************************************************* [ 0.277024] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.280023] ** ** [ 0.283034] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.285016] ** ** [ 0.287017] ** This means that this kernel is built to expose internal ** [ 0.290026] ** IOMMU data structures, which may compromise security on ** [ 0.292016] ** your system. ** [ 0.295025] ** ** [ 0.297018] ** If you see this message and you are not debugging the ** [ 0.300032] ** kernel, report this immediately to your vendor! ** [ 0.302015] ** ** [ 0.304019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.307020] ************************************************************* [ 0.309746] NET: Registered protocol family 16 [ 0.311444] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.314086] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.316080] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.319169] cpuidle: using governor menu [ 0.320686] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.323075] PCI: Using configuration type 1 for base access [ 0.325231] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.336182] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.339112] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.342079] cryptd: max_cpu_qlen set to 1000 [ 0.345077] ACPI: Added _OSI(Module Device) [ 0.347024] ACPI: Added _OSI(Processor Device) [ 0.349019] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.351018] ACPI: Added _OSI(Processor Aggregator Device) [ 0.355226] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.363073] ACPI: Interpreter enabled [ 0.365070] ACPI: PM: (supports S0 S3 S4 S5) [ 0.367015] ACPI: Using IOAPIC for interrupt routing [ 0.368179] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.371463] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.381424] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.384060] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.386030] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.390114] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.395460] acpiphp: Slot [2] registered [ 0.396131] acpiphp: Slot [5] registered [ 0.398212] acpiphp: Slot [6] registered [ 0.400237] acpiphp: Slot [7] registered [ 0.402347] acpiphp: Slot [8] registered [ 0.404201] acpiphp: Slot [9] registered [ 0.405289] acpiphp: Slot [10] registered [ 0.407199] acpiphp: Slot [3] registered [ 0.409155] acpiphp: Slot [4] registered [ 0.411170] acpiphp: Slot [11] registered [ 0.413110] acpiphp: Slot [12] registered [ 0.414108] acpiphp: Slot [13] registered [ 0.416125] acpiphp: Slot [14] registered [ 0.417104] acpiphp: Slot [15] registered [ 0.419148] acpiphp: Slot [16] registered [ 0.421124] acpiphp: Slot [17] registered [ 0.423123] acpiphp: Slot [18] registered [ 0.425094] acpiphp: Slot [19] registered [ 0.426115] acpiphp: Slot [20] registered [ 0.428118] acpiphp: Slot [21] registered [ 0.429092] acpiphp: Slot [22] registered [ 0.432176] acpiphp: Slot [23] registered [ 0.434140] acpiphp: Slot [24] registered [ 0.435122] acpiphp: Slot [25] registered [ 0.438199] acpiphp: Slot [26] registered [ 0.440149] acpiphp: Slot [27] registered [ 0.442129] acpiphp: Slot [28] registered [ 0.443176] acpiphp: Slot [29] registered [ 0.445164] acpiphp: Slot [30] registered [ 0.447163] acpiphp: Slot [31] registered [ 0.448089] PCI host bridge to bus 0000:00 [ 0.450044] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.453039] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.457032] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.460037] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.463025] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.466034] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.469179] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.473165] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.476440] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.486017] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.491640] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.494025] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.497022] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.500029] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.503746] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.507958] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.511054] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.514911] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.521019] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.533854] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.539016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.545327] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.552021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.560022] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.575020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.588138] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.595016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.602024] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.625020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.636294] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.643015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.648022] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.662026] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.671281] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.678018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.689017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.705016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.716000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.721020] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.726021] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.741021] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.752871] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.758023] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.765025] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.780018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.792709] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.795476] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.798388] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.801492] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.804232] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.809159] iommu: Default domain type: Passthrough [ 0.812741] SCSI subsystem initialized [ 0.815154] ACPI: bus type USB registered [ 0.816199] usbcore: registered new interface driver usbfs [ 0.818093] usbcore: registered new interface driver hub [ 0.821103] usbcore: registered new device driver usb [ 0.823225] pps_core: LinuxPPS API ver. 1 registered [ 0.825014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.828065] PTP clock support registered [ 0.830281] EDAC MC: Ver: 3.0.0 [ 0.833082] PCI: Using ACPI for IRQ routing [ 0.834530] NetLabel: Initializing [ 0.836013] NetLabel: domain hash size = 128 [ 0.838010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.840088] NetLabel: unlabeled traffic allowed by default [ 0.843257] vgaarb: loaded [ 0.844436] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.846017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.854000] clocksource: Switched to clocksource kvm-clock [ 0.996203] VFS: Disk quotas dquot_6.6.0 [ 0.999155] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.003289] *** VALIDATE ramfs *** [ 1.004945] *** VALIDATE hugetlbfs *** [ 1.006704] pnp: PnP ACPI init [ 1.009031] pnp: PnP ACPI: found 6 devices [ 1.026668] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.030553] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.032751] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.036699] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.039600] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.042494] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.046125] NET: Registered protocol family 2 [ 1.049328] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.054855] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.059489] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.066583] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.071409] TCP: Hash tables configured (established 65536 bind 65536) [ 1.075527] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.079536] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.083213] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.086796] NET: Registered protocol family 1 [ 1.091085] RPC: Registered named UNIX socket transport module. [ 1.093843] RPC: Registered udp transport module. [ 1.095975] RPC: Registered tcp transport module. [ 1.098044] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.100977] NET: Registered protocol family 44 [ 1.103263] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.105802] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.108360] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.111239] PCI: CLS 0 bytes, default 64 [ 1.113313] Unpacking initramfs... [ 2.536743] debug: unmapping init [mem 0xffff8ad77cc54000-0xffff8ad77ffbffff] [ 2.540335] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.542523] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.545177] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.052263] Initialise system trusted keyrings [ 3.055058] Key type blacklist registered [ 3.056715] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.066650] zbud: loaded [ 3.069955] *** VALIDATE nfs *** [ 3.071639] *** VALIDATE nfs4 *** [ 3.073544] pstore: using deflate compression [ 3.077599] Platform Keyring initialized [ 3.213920] NET: Registered protocol family 38 [ 3.216759] Key type asymmetric registered [ 3.218828] Asymmetric key parser 'x509' registered [ 3.221057] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.224966] io scheduler mq-deadline registered [ 3.227034] io scheduler kyber registered [ 3.229603] io scheduler bfq registered [ 3.231142] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.235532] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.238832] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.242607] ACPI: Power Button [PWRF] [ 3.251378] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.259447] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.274919] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.281693] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.295871] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.324155] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.352882] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.358784] Non-volatile memory driver v1.3 [ 3.360567] Linux agpgart interface v0.103 [ 3.408713] virtio_blk virtio1: [vda] 68000 512-byte logical blocks (34.8 MB/33.2 MiB) [ 3.413666] vda: detected capacity change from 0 to 34816000 [ 3.435374] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.438883] vdb: detected capacity change from 0 to 1073741824 [ 3.455638] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.459220] vdc: detected capacity change from 0 to 2621440000 [ 3.479384] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.482262] vdd: detected capacity change from 0 to 2621440000 [ 3.496799] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.499592] vde: detected capacity change from 0 to 4294967296 [ 3.513403] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.515911] vdf: detected capacity change from 0 to 4294967296 [ 3.522220] libphy: Fixed MDIO Bus: probed [ 3.528952] usbcore: registered new interface driver usbserial_generic [ 3.532191] usbserial: USB Serial support registered for generic [ 3.535087] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.540159] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.542382] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.545089] mousedev: PS/2 mouse device common for all mice [ 3.548432] rtc_cmos 00:05: RTC can wake from S4 [ 3.550440] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.553244] rtc_cmos 00:05: registered as rtc0 [ 3.556601] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.559887] intel_pstate: CPU model not supported [ 3.563158] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.565537] hid: raw HID events driver (C) Jiri Kosina [ 3.569816] usbcore: registered new interface driver usbhid [ 3.571758] usbhid: USB HID core driver [ 3.573344] drop_monitor: Initializing network drop monitor service [ 3.573415] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.575624] Initializing XFRM netlink socket [ 3.582104] NET: Registered protocol family 10 [ 3.585412] Segment Routing with IPv6 [ 3.586991] NET: Registered protocol family 17 [ 3.589385] mpls_gso: MPLS GSO support [ 3.595472] RAS: Correctable Errors collector initialized. [ 3.597990] AVX version of gcm_enc/dec engaged. [ 3.599932] AES CTR mode by8 optimization enabled [ 3.681798] sched_clock: Marking stable (3681778724, 0)->(4644075849, -962297125) [ 3.685334] registered taskstats version 1 [ 3.687183] Loading compiled-in X.509 certificates [ 3.688935] zswap: loaded using pool lzo/zbud [ 3.716853] Key type big_key registered [ 3.727369] Key type encrypted registered [ 3.729186] ima: No TPM chip found, activating TPM-bypass! [ 3.732629] ima: Allocated hash algorithm: sha1 [ 3.734532] ima: No architecture policies found [ 3.736704] evm: Initialising EVM extended attributes: [ 3.738654] evm: security.selinux [ 3.740049] evm: security.ima [ 3.741761] evm: security.capability [ 3.743153] evm: HMAC attrs: 0x1 [ 3.745422] rtc_cmos 00:05: setting system clock to 2026-04-14 20:56:14 UTC (1776200174) [ 3.751166] debug: unmapping init [mem 0xffffffffa9003000-0xffffffffa91fffff] [ 3.754169] debug: unmapping init [mem 0xffffffffa7d82000-0xffffffffa8058fff] [ 3.767418] Write protecting the kernel read-only data: 28672k [ 3.770465] debug: unmapping init [mem 0xffffffffa6403000-0xffffffffa65fffff] [ 3.773504] debug: unmapping init [mem 0xffffffffa6d14000-0xffffffffa6dfffff] [ 3.808786] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.818814] systemd[1]: Detected virtualization kvm. [ 3.820448] systemd[1]: Detected architecture x86-64. [ 3.821988] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.844398] systemd[1]: No hostname configured. [ 3.846657] systemd[1]: Set hostname to . [ 3.848955] random: systemd: uninitialized urandom read (16 bytes read) [ 3.852239] systemd[1]: Initializing machine ID from random generator. [ 3.989472] random: systemd: uninitialized urandom read (16 bytes read) [ 3.993142] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 4.000042] random: systemd: uninitialized urandom read (16 bytes read) [ 4.002681] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 4.006233] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.662486] device-mapper: uevent: version 1.0.3 [ 4.665178] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.445617] virtio_net virtio0 ens2: renamed from eth0 [ 5.476900] scsi host0: ata_piix [ 5.487250] random: fast init done [ 5.513521] scsi host1: ata_piix [ 5.515073] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.518676] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.848735] dracut-initqueue[604]: RTNETLINK answers: File exists [ 10.248578] random: crng init done [ 10.250298] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.672503] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.845174] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.111481] SELinux: Disabled at runtime. [ 12.170800] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.182258] systemd[1]: Detected virtualization kvm. [ 12.184404] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.700463] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.704054] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.709183] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.714236] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.719347] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.731964] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.738248] systemd[1]: Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Kernel Socket. Mounting POSIX Message Queue File System... [ OK ] Created slice system-getty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on udev Control Socket. [ OK ] Reached target RPC Port Mapper. Mounting Huge Pages File System... [ OK ] Stopped target Initrd Root File System. Starting udev Coldplug all Devices... [ OK ] Listening on Process Core Dump Socket. Mounting Kernel Debug File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. Activating swap /dev/disk/by-label/SWAP... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target rpc_pipefs.target. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Remount Root and Kernel File Systems... [ 12.904777] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on initctl Compatibility Named Pipe. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 13.202226] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.528667] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.613032] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.707139] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.725364] EDAC sbridge: Ver: 1.1.2 [ 15.232375] Key type dns_resolver registered [ 15.576582] NFS: Registering the id_resolver key type [ 15.579179] Key type id_resolver registered [ 15.581124] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg444-server login: [ 38.146180] libcfs: loading out-of-tree module taints kernel. [ 38.168119] alg: No test for adler32 (adler32-zlib) [ 38.918474] Key type ._llcrypt registered [ 38.920699] Key type .llcrypt registered [ 38.974934] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 59.603951] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 61.009405] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 61.776811] Lustre: Lustre: Build Version: 2.15.8_1_g00e7d23 [ 62.837737] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 62.845782] LNet: Accept secure, port 988 [ 64.680777] Key type lgssc registered [ 66.173851] Lustre: Echo OBD driver; http://www.lustre.org/ [ 80.790173] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 83.797440] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 93.345687] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 100.965499] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 108.234451] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 122.728085] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 136.063635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 136.241817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 137.703989] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 137.755926] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 137.891520] Lustre: lustre-MDT0000: new disk, initializing [ 138.055944] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 138.108663] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 143.739523] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 157.999313] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 158.086901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 158.180312] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 158.213521] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 158.220791] Lustre: Skipped 1 previous similar message [ 158.322405] Lustre: lustre-MDT0001: new disk, initializing [ 158.393775] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 158.441626] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 158.458676] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 162.072953] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 174.303837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 174.376127] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 174.381612] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 174.568604] Lustre: lustre-OST0000: new disk, initializing [ 174.572292] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 174.624531] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 179.111657] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 182.281049] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 182.292495] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 191.627972] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 191.696194] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 191.703374] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 191.825246] Lustre: lustre-OST0001: new disk, initializing [ 191.832837] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 191.897736] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 196.326075] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 200.753241] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 200.773638] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 205.773888] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 211.432566] hrtimer: interrupt took 8968158 ns [ 211.841813] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 222.441259] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing check_logdir /tmp/testlogs/ [ 229.810604] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing yml_node [ 235.838342] Lustre: DEBUG MARKER: Client: 2.15.8.1 [ 239.267975] Lustre: DEBUG MARKER: MDS: 2.15.8.1 [ 242.274220] Lustre: DEBUG MARKER: OSS: 2.15.8.1 [ 244.195967] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Apr 14 17:00:13 EDT 2026 [ 255.124607] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball 110 [ 256.915544] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 [ 276.449213] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 276.458090] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 276.483564] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 277.473350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 277.474663] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 279.348268] Lustre: server umount lustre-MDT0000 complete [ 282.593333] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 282.614566] LustreError: Skipped 3 previous similar messages [ 283.819308] LustreError: 6259:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776200454 with bad export cookie 17025449173332274468 [ 283.830672] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 283.832158] LustreError: 6259:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 287.714410] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 287.715361] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 287.744351] LustreError: Skipped 1 previous similar message [ 287.770302] Lustre: Skipped 2 previous similar messages [ 287.782292] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 287.790449] Lustre: Skipped 4 previous similar messages [ 288.744684] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 288.771362] LustreError: Skipped 1 previous similar message [ 290.259826] Lustre: server umount lustre-MDT0001 complete [ 295.344903] Lustre: server umount lustre-OST0000 complete [ 300.851025] Lustre: server umount lustre-OST0001 complete [ 309.500616] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 323.031593] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 334.278053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 341.169166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 347.529491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 354.064423] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 361.015906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 361.104303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 361.367715] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 361.403544] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 361.501996] Lustre: lustre-MDT0000: new disk, initializing [ 361.615637] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 361.635285] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 366.141896] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 377.405513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 377.515542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 377.582502] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 377.616664] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 377.621814] Lustre: Skipped 1 previous similar message [ 377.702793] Lustre: lustre-MDT0001: new disk, initializing [ 377.820415] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 377.856491] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 377.866771] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 382.764761] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 394.659825] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 401.930311] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 409.388114] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 409.443479] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 409.449228] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 409.757550] Lustre: lustre-OST0000: new disk, initializing [ 409.762193] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 409.860798] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 411.230978] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 411.239765] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 415.133926] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 425.980292] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 431.473752] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 431.709458] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 437.430820] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 437.861571] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 442.339704] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 442.341512] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 442.345711] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 442.345721] Lustre: Skipped 4 previous similar messages [ 442.383820] Lustre: Skipped 2 previous similar messages [ 446.343233] Lustre: server umount lustre-OST0000 complete [ 452.584654] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 452.588881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 452.605349] Lustre: Skipped 1 previous similar message [ 452.620339] Lustre: Skipped 2 previous similar messages [ 456.496398] Lustre: server umount lustre-MDT0000 complete [ 457.700204] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 457.725346] LustreError: Skipped 2 previous similar messages [ 461.264919] LustreError: 14366:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776200632 with bad export cookie 17025449173332275924 [ 461.267942] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 461.281785] LustreError: 14366:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 466.050975] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 17:03:54 (1776200634) [ 468.616361] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 470.925868] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 17:03:59 (1776200639) [ 477.651745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 478.150338] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 478.165463] LustreError: Skipped 3 previous similar messages [ 478.253611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 483.184039] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 488.418046] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 488.435198] LustreError: Skipped 1 previous similar message [ 491.810566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 497.426103] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 507.125546] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 513.861576] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 521.344438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 527.525704] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 536.130975] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 544.738731] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 544.743490] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 544.761567] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 544.772685] LustreError: Skipped 1 previous similar message [ 550.066940] Lustre: server umount lustre-OST0000 complete [ 550.074830] Lustre: Skipped 1 previous similar message [ 563.909966] LustreError: 18904:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776200734 with bad export cookie 17025449173332276778 [ 563.913584] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 563.919885] LustreError: 18904:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 577.539472] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 580.231673] Key type lgssc unregistered [ 580.575313] LNet: 22580:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 580.611267] LNet: Removed LNI 192.168.204.144@tcp [ 581.630675] Key type .llcrypt unregistered [ 581.635093] Key type ._llcrypt unregistered [ 610.965580] alg: No test for adler32 (adler32-zlib) [ 611.734280] Key type ._llcrypt registered [ 611.735832] Key type .llcrypt registered [ 611.936931] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 614.349360] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 615.116427] Lustre: Lustre: Build Version: 2.15.8_1_g00e7d23 [ 615.333720] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 615.341604] LNet: Accept secure, port 988 [ 617.009082] Key type lgssc registered [ 618.241519] Lustre: Echo OBD driver; http://www.lustre.org/ [ 627.844863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 629.489859] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 629.559057] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 634.281926] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 634.853109] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 639.985991] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 645.091129] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 645.664367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 645.969150] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 651.788776] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 660.948549] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 667.166857] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 676.365681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 676.797326] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 681.913226] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 681.992286] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 691.034930] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 706.538235] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 706.549161] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 706.560178] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 711.654773] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 711.666929] Lustre: Skipped 1 previous similar message [ 711.894537] Lustre: server umount lustre-OST0000 complete [ 716.773163] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 716.797873] Lustre: Skipped 1 previous similar message [ 716.807457] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 716.810363] Lustre: Skipped 1 previous similar message [ 721.889663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 721.906278] Lustre: Skipped 1 previous similar message [ 722.255405] Lustre: server umount lustre-MDT0000 complete [ 726.613814] LustreError: 24051:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776200897 with bad export cookie 12555578359645897595 [ 726.622407] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 726.630751] LustreError: 24051:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 727.071558] Lustre: server umount lustre-MDT0001 complete [ 740.465687] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 743.790476] Key type lgssc unregistered [ 744.225900] LNet: 27748:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 744.240381] LNet: Removed LNI 192.168.204.144@tcp [ 745.089747] Key type .llcrypt unregistered [ 745.095463] Key type ._llcrypt unregistered [ 770.111830] alg: No test for adler32 (adler32-zlib) [ 770.866968] Key type ._llcrypt registered [ 770.871573] Key type .llcrypt registered [ 771.019778] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 771.976528] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 773.057300] Lustre: Lustre: Build Version: 2.15.8_1_g00e7d23 [ 773.335943] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 773.344574] LNet: Accept secure, port 988 [ 775.064137] Key type lgssc registered [ 776.289691] Lustre: Echo OBD driver; http://www.lustre.org/ [ 784.309513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 785.832585] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 785.920809] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 791.012846] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 791.117717] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 796.134435] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 800.456820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 800.894216] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 805.915756] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 814.077462] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 820.549708] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 826.796145] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 827.168660] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 831.956629] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 836.467792] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 840.014946] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 846.728929] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.44@tcp (stopping) [ 846.835138] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 846.835493] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 848.004441] Lustre: server umount lustre-MDT0000 complete [ 851.827908] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 852.306884] LustreError: 31876:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776201023 with bad export cookie 11798977124837394761 [ 852.311307] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 852.317079] LustreError: 31876:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 856.939234] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 856.964624] LustreError: Skipped 3 previous similar messages [ 856.979824] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.44@tcp (stopping) [ 856.988824] Lustre: Skipped 3 previous similar messages [ 857.060436] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 857.086371] Lustre: Skipped 2 previous similar messages [ 858.838913] Lustre: server umount lustre-MDT0001 complete [ 880.725336] Lustre: server umount lustre-OST0000 complete [ 898.452883] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 901.088580] Key type lgssc unregistered [ 901.379067] LNet: 33209:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 901.395620] LNet: Removed LNI 192.168.204.144@tcp [ 902.169324] Key type .llcrypt unregistered [ 902.172554] Key type ._llcrypt unregistered [ 923.625500] alg: No test for adler32 (adler32-zlib) [ 924.385436] Key type ._llcrypt registered [ 924.389391] Key type .llcrypt registered [ 924.522161] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 925.688065] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 926.060393] Lustre: Lustre: Build Version: 2.15.8_1_g00e7d23 [ 926.254802] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 926.262242] LNet: Accept secure, port 988 [ 927.905171] Key type lgssc registered [ 928.944180] Lustre: Echo OBD driver; http://www.lustre.org/ [ 936.511326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 937.982112] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 938.082706] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 942.712238] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 943.081055] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 948.210594] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 952.290644] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 953.296659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 953.663336] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 958.966972] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 968.715749] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 976.123417] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 983.025770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 983.426801] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 987.514621] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 988.255837] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 996.867685] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1003.921062] Lustre: lustre-OST0000: Not available for connect from 192.168.204.44@tcp (stopping) [ 1005.028350] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1005.039134] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1005.066418] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1008.098071] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1008.099290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1008.110358] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1008.136551] Lustre: Skipped 1 previous similar message [ 1009.405751] Lustre: server umount lustre-OST0000 complete [ 1028.588186] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1028.613798] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1028.624957] Lustre: Skipped 2 previous similar messages [ 1033.700291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1033.709048] Lustre: Skipped 1 previous similar message [ 1041.376344] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1041.506597] Lustre: server umount lustre-MDT0000 complete [ 1043.938877] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1043.959205] LustreError: Skipped 1 previous similar message [ 1045.301670] LustreError: 34656:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776201216 with bad export cookie 16766265522434672745 [ 1045.329457] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1051.735518] Lustre: server umount lustre-MDT0001 complete [ 1064.342827] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 1068.126148] Key type lgssc unregistered [ 1068.468421] LNet: 38507:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1068.479510] LNet: Removed LNI 192.168.204.144@tcp [ 1069.194828] Key type .llcrypt unregistered [ 1069.202151] Key type ._llcrypt unregistered [ 1094.250194] alg: No test for adler32 (adler32-zlib) [ 1095.013623] Key type ._llcrypt registered [ 1095.015389] Key type .llcrypt registered [ 1095.157399] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 1096.133361] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1096.654599] Lustre: Lustre: Build Version: 2.15.8_1_g00e7d23 [ 1096.859192] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 1096.862625] LNet: Accept secure, port 988 [ 1098.560288] Key type lgssc registered [ 1099.724737] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1108.577315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1110.185780] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1110.278507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1114.283574] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1115.284511] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 1119.718898] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1124.804846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1124.839131] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1125.189049] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1129.723400] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 1138.117089] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1144.935655] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1151.225159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1151.604853] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1156.032972] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 1159.795606] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 1163.406943] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1169.085191] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1172.931513] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1181.169505] LNetError: 120-3: Refusing connection from 192.168.204.144 for 127.0.0.2@tcp: No matching NI [ 1181.180707] LNetError: 39273:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1181.207177] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1185.645213] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-04-1417h15m53s [ 1187.345973] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1190.801551] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.44@tcp (stopping) [ 1190.885459] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1190.906213] Lustre: Skipped 2 previous similar messages [ 1190.914280] LNetError: 120-3: Refusing connection from 192.168.204.144 for 127.0.0.2@tcp: No matching NI [ 1190.929020] LNetError: 39270:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1190.945623] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1195.884077] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.44@tcp (stopping) [ 1195.891893] Lustre: Skipped 3 previous similar messages [ 1196.006054] LNetError: 120-3: Refusing connection from 192.168.204.144 for 127.0.0.2@tcp: No matching NI [ 1196.017655] LNetError: 39271:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1196.031676] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1200.034715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1200.048948] Lustre: Skipped 5 previous similar messages [ 1204.192147] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1204.303823] Lustre: server umount lustre-MDT0000 complete [ 1205.220440] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1205.223682] LNetError: 120-3: Refusing connection from 192.168.204.144 for 127.0.0.2@tcp: No matching NI [ 1205.253603] LNetError: 39272:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1205.266454] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1208.077449] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1211.360260] Lustre: 39282:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776201375/real 1776201375] req@000000007abbc58c x1862482019766144/t0(0) o400->MGC192.168.204.144@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776201382 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1211.395663] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1213.273475] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1213.291685] LustreError: Skipped 10 previous similar messages [ 1214.512900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1216.489532] LNetError: 120-3: Refusing connection from 192.168.204.144 for 127.0.0.2@tcp: No matching NI [ 1216.498272] LNetError: 39273:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1216.513828] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1217.578979] Lustre: Evicted from MGS (at 192.168.204.144@tcp) after server handle changed from 0x3cc5a721d091fec6 to 0x3cc5a721d092031f [ 1217.598320] Lustre: MGC192.168.204.144@tcp: Connection restored to 192.168.204.144@tcp (at 0@lo) [ 1217.860145] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1217.923850] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:193 [ 1223.144027] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1223.158334] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to (at 0@lo) [ 1223.173982] Lustre: Skipped 1 previous similar message [ 1223.282815] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 1226.686925] Lustre: DEBUG MARKER: Wait for df (28939) ... [ 1228.484670] Lustre: DEBUG MARKER: done [ 1237.986980] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1238.000691] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1238.022468] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1238.036411] Lustre: Skipped 1 previous similar message [ 1238.513195] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1243.628847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1243.643687] Lustre: Skipped 3 previous similar messages [ 1248.224201] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1248.350944] Lustre: server umount lustre-OST0000 complete [ 1253.868634] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1253.869962] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1253.885487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1253.899112] Lustre: Skipped 2 previous similar messages [ 1253.910450] LNetError: 120-3: Refusing connection from 192.168.204.144 for 127.0.0.2@tcp: No matching NI [ 1253.911860] Lustre: Skipped 1 previous similar message [ 1253.938812] LNetError: 39270:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1253.953259] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1258.747412] Lustre: server umount lustre-MDT0000 complete [ 1258.978884] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1259.006836] LustreError: Skipped 7 previous similar messages [ 1262.566866] LustreError: 42668:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776201433 with bad export cookie 4379089976362664735 [ 1262.567896] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1262.576491] LustreError: 42668:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1262.859060] Lustre: server umount lustre-MDT0001 complete [ 1275.028775] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 1278.205559] Key type lgssc unregistered [ 1278.470664] LNet: 45075:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1278.478394] LNet: Removed LNI 192.168.204.144@tcp [ 1279.333490] Key type .llcrypt unregistered [ 1279.338857] Key type ._llcrypt unregistered [ 1316.911230] alg: No test for adler32 (adler32-zlib) [ 1317.677725] Key type ._llcrypt registered [ 1317.680525] Key type .llcrypt registered [ 1317.803839] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 1330.788290] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 1331.773885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1332.104112] Lustre: Lustre: Build Version: 2.15.8_1_g00e7d23 [ 1332.321178] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 1332.327464] LNet: Accept secure, port 988 [ 1333.980950] Key type lgssc registered [ 1335.136912] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1344.334694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1351.176405] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1358.353351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1365.114537] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1371.379868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1371.447443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1372.706132] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1372.773169] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1372.902633] Lustre: lustre-MDT0000: new disk, initializing [ 1373.078848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1373.105265] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1377.969821] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 1389.264845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1389.335607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1389.420095] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1389.452754] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1389.456810] Lustre: Skipped 1 previous similar message [ 1389.525731] Lustre: lustre-MDT0001: new disk, initializing [ 1389.605135] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1389.634713] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1389.644301] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1394.042742] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 1404.063785] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1410.630877] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1416.963856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1417.030935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1417.040679] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1417.258750] Lustre: lustre-OST0000: new disk, initializing [ 1417.265944] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1417.337589] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1421.759244] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 1422.902180] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1422.914878] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1431.346503] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1437.218195] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1437.501375] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1443.636331] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 1443.921469] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1448.932689] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1448.947291] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1448.970510] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1452.386524] Lustre: server umount lustre-OST0000 complete [ 1459.174476] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1459.181652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1459.196360] Lustre: Skipped 2 previous similar messages [ 1459.206760] Lustre: Skipped 2 previous similar messages [ 1462.822658] Lustre: server umount lustre-MDT0000 complete [ 1464.302518] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1464.320606] LustreError: Skipped 1 previous similar message [ 1465.316064] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1465.324087] LustreError: Skipped 1 previous similar message [ 1467.092761] LustreError: 49171:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776201637 with bad export cookie 17021808891680790970 [ 1467.095185] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1467.101771] LustreError: 49171:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1467.420481] Lustre: server umount lustre-MDT0001 complete [ 1477.168905] Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 17:20:46 (1776201646) [ 1484.260970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1484.731541] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1484.899704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1489.891078] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1490.034496] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 1495.012412] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1498.363274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1498.754492] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1504.096582] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 1513.249479] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1519.770290] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1525.907309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1526.185771] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1531.340984] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 1539.941578] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1546.621466] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-04-1417h21m55s [ 1548.379789] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1555.650411] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1555.955588] LNetError: 120-3: Refusing connection from 192.168.204.144 for 127.0.0.2@tcp: No matching NI [ 1555.965781] LNetError: 47383:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1555.971121] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1558.459352] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1558.530843] LustreError: 53726:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1578.906205] Lustre: lustre-MDT0000: Client 17d38b74-e237-4ae3-83c1-d3808436c279 (at 192.168.204.44@tcp) reconnecting [ 1598.544380] LustreError: 53726:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 136 awake [ 1601.146468] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1603.997755] Lustre: DEBUG MARKER: done [ 1616.384518] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1616.400022] LustreError: Skipped 1 previous similar message [ 1616.408571] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1616.416616] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1616.422655] Lustre: Skipped 1 previous similar message [ 1616.473033] Lustre: Skipped 1 previous similar message [ 1617.809859] Lustre: server umount lustre-OST0000 complete [ 1626.593822] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1626.618565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1626.625370] Lustre: Skipped 1 previous similar message [ 1626.630719] LNetError: 120-3: Refusing connection from 192.168.204.144 for 127.0.0.2@tcp: No matching NI [ 1626.636861] LNetError: Skipped 1 previous similar message [ 1626.640836] LNetError: 47386:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1626.646837] LNetError: 47386:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Skipped 1 previous similar message [ 1626.657519] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1626.671245] LNetError: Skipped 1 previous similar message [ 1629.070598] Lustre: server umount lustre-MDT0000 complete [ 1631.720285] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1631.735735] LNetError: 120-3: Refusing connection from 192.168.204.144 for 127.0.0.2@tcp: No matching NI [ 1631.741338] LustreError: Skipped 1 previous similar message [ 1631.769234] LNetError: 47385:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1631.775903] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1633.984426] LustreError: 56522:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776201804 with bad export cookie 17021808891680791824 [ 1633.985834] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1634.009990] LustreError: 56522:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1651.275750] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 1654.933716] Key type lgssc unregistered [ 1655.333844] LNet: 58151:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1655.354262] LNet: Removed LNI 192.168.204.144@tcp [ 1656.422498] Key type .llcrypt unregistered [ 1656.425200] Key type ._llcrypt unregistered [ 1694.188259] alg: No test for adler32 (adler32-zlib) [ 1694.942558] Key type ._llcrypt registered [ 1694.946208] Key type .llcrypt registered [ 1695.167067] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 1710.440944] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 1711.784387] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1712.244926] Lustre: Lustre: Build Version: 2.15.8_1_g00e7d23 [ 1712.532632] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 1712.540654] LNet: Accept secure, port 988 [ 1714.312178] Key type lgssc registered [ 1715.679609] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1724.936527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1731.901530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1739.030864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1746.607607] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1754.239435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1754.313906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1755.628558] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1755.653382] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1755.720282] Lustre: lustre-MDT0000: new disk, initializing [ 1755.809875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1755.835271] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1760.977596] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 1772.005944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1772.094867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1772.198842] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1772.298340] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1772.302954] Lustre: Skipped 1 previous similar message [ 1772.418709] Lustre: lustre-MDT0001: new disk, initializing [ 1772.552470] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1772.618976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1772.636438] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1778.118256] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 1789.850314] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1796.872762] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1804.089537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1804.147101] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1804.155556] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1804.426037] Lustre: lustre-OST0000: new disk, initializing [ 1804.437278] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1804.528228] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1809.291766] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 1812.531931] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1812.549073] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1817.494570] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1822.077290] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1822.324246] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1826.083656] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 1826.273804] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1828.321132] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1828.321577] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1828.334417] LustreError: Skipped 1 previous similar message [ 1828.336760] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1828.355411] Lustre: Skipped 1 previous similar message [ 1833.449581] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1833.454389] Lustre: Skipped 2 previous similar messages [ 1834.218487] Lustre: server umount lustre-OST0000 complete [ 1838.571352] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1838.572770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1838.586511] Lustre: Skipped 1 previous similar message [ 1843.682693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1843.689055] Lustre: Skipped 2 previous similar messages [ 1848.804321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1848.814794] Lustre: Skipped 1 previous similar message [ 1852.384136] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1852.519348] Lustre: server umount lustre-MDT0000 complete [ 1853.922508] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1853.940908] LustreError: Skipped 1 previous similar message [ 1854.952572] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1854.968621] LustreError: Skipped 1 previous similar message [ 1856.343287] LustreError: 62267:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776202027 with bad export cookie 10380732443879791678 [ 1856.346707] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1856.354338] LustreError: 62267:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1856.611819] Lustre: server umount lustre-MDT0001 complete [ 1864.477377] Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 17:27:13 (1776202033) [ 1866.084260] Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST [ 1867.853502] Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 17:27:16 (1776202036) [ 1878.997935] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 1883.328062] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1884.062184] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1884.132437] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1885.412402] Lustre: Failing over lustre-MDT0000 [ 1885.418215] LustreError: 67290:0:(osp_object.c:629:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1885.436551] LustreError: 67290:0:(lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1885.445678] LustreError: 67290:0:(lod_dev.c:424:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000 get update log failed: rc = -5 [ 1891.808396] Lustre: 67341:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776202056/real 1776202056] req@00000000ac5fd33c x1862482665690112/t0(0) o251->MGC192.168.204.144@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776202062 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1892.029624] Lustre: server umount lustre-MDT0000 complete [ 1901.427891] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 17:27:49 (1776202069) [ 1908.436910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1908.763538] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1908.855966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1913.006093] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 1914.339864] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1921.635187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1921.898167] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1926.758286] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 1934.306390] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1940.551110] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1947.540321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1947.775296] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1952.031935] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 1962.335827] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1967.985851] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1978.337318] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1978.343727] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1978.363651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1980.900277] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1980.914635] Lustre: Skipped 1 previous similar message [ 1981.114228] Lustre: server umount lustre-MDT0000 complete [ 1985.198946] LustreError: 67890:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776202155 with bad export cookie 10380732443879792805 [ 1985.203369] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1985.213351] LustreError: 67890:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1989.384608] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1993.121743] Lustre: 60494:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776202156/real 1776202156] req@0000000032453134 x1862482665709376/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776202163 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1993.160202] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1995.298979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1997.235654] Lustre: 60494:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776202160/real 1776202160] req@000000008229b6c6 x1862482665709440/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776202167 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1997.284310] Lustre: 60494:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2004.185256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2017.767164] Lustre: Evicted from MGS (at 192.168.204.144@tcp) after server handle changed from 0x900fc52881d3e0a5 to 0x900fc52881d3e838 [ 2017.775907] Lustre: MGC192.168.204.144@tcp: Connection restored to 192.168.204.144@tcp (at 0@lo) [ 2017.958744] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2017.984738] LustreError: Skipped 1 previous similar message [ 2018.056715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2022.498428] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 2032.603700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2033.001285] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:65 [ 2034.031419] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 2037.267506] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 2045.035862] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2051.139902] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2058.723508] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2058.733581] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2058.754433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2058.761754] Lustre: Skipped 3 previous similar messages [ 2064.868102] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2064.877847] LustreError: Skipped 6 previous similar messages [ 2064.913453] Lustre: server umount lustre-MDT0000 complete [ 2064.916401] Lustre: Skipped 1 previous similar message [ 2068.667700] LustreError: 67891:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776202239 with bad export cookie 10380732443879794744 [ 2068.670244] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2068.697608] LustreError: 67891:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2077.089232] Lustre: 60493:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776202240/real 1776202240] req@00000000e2b88fc9 x1862482665724416/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776202247 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2077.133891] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2077.164101] Lustre: Skipped 2 previous similar messages [ 2079.183890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2081.248233] Lustre: 60493:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776202244/real 1776202244] req@0000000013f457a2 x1862482665724480/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776202251 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2081.300176] Lustre: 60493:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2092.102807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2101.735128] Lustre: Evicted from MGS (at 192.168.204.144@tcp) after server handle changed from 0x900fc52881d3e838 to 0x900fc52881d3ee4a [ 2101.747335] Lustre: MGC192.168.204.144@tcp: Connection restored to (at 0@lo) [ 2101.749820] Lustre: Skipped 1 previous similar message [ 2101.991547] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2102.006423] LustreError: Skipped 2 previous similar messages [ 2102.076212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2102.081159] Lustre: Skipped 1 previous similar message [ 2102.154914] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 2107.206368] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 2107.849137] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 2117.460435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2118.078872] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:97 [ 2123.243215] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 2124.149522] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 2131.715526] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2137.478353] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2148.852409] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2148.860065] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2148.860472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2148.860476] Lustre: Skipped 2 previous similar messages [ 2148.866249] Lustre: Skipped 1 previous similar message [ 2151.776938] Lustre: server umount lustre-MDT0000 complete [ 2151.786259] Lustre: Skipped 1 previous similar message [ 2155.580500] LustreError: 67892:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776202326 with bad export cookie 10380732443879796298 [ 2155.582863] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2155.597818] LustreError: 67892:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2163.900254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2166.243109] Lustre: 60496:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776202329/real 1776202329] req@00000000ac35faf3 x1862482665738368/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776202336 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2167.442459] Lustre: DEBUG MARKER: files compared the same [ 2187.090638] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 2189.822444] Key type lgssc unregistered [ 2190.082843] LNet: 77956:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2190.100360] LNet: Removed LNI 192.168.204.144@tcp [ 2190.965682] Key type .llcrypt unregistered [ 2190.969554] Key type ._llcrypt unregistered [ 2212.030927] alg: No test for adler32 (adler32-zlib) [ 2212.789441] Key type ._llcrypt registered [ 2212.795358] Key type .llcrypt registered [ 2212.928413] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 2214.021183] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2214.509415] Lustre: Lustre: Build Version: 2.15.8_1_g00e7d23 [ 2214.737759] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 2214.743777] LNet: Accept secure, port 988 [ 2216.410669] Key type lgssc registered [ 2217.526808] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2225.537473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2227.036305] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2227.092232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2232.294141] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2232.324298] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug +malloc all 8 [ 2237.412749] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2241.333947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2241.638316] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2245.856682] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug +malloc all 8 [ 2252.946208] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2260.346789] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2269.753776] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2270.543476] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2271.555315] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:129 [ 2277.175436] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug +malloc all 8 [ 2277.700863] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 2286.028427] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2297.315292] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2297.319748] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2297.331955] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2298.340126] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2298.341051] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2298.341228] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2298.345955] Lustre: Skipped 1 previous similar message [ 2299.911477] Lustre: server umount lustre-OST0000 complete [ 2308.076960] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2308.083915] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2308.104512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2309.920447] Lustre: server umount lustre-MDT0000 complete [ 2313.340525] LustreError: 79399:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776202484 with bad export cookie 242234033087315820 [ 2313.349556] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2313.562697] Lustre: server umount lustre-MDT0001 complete [ 2325.986910] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 2328.932918] Key type lgssc unregistered [ 2329.336198] LNet: 83074:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2329.368951] LNet: Removed LNI 192.168.204.144@tcp [ 2330.106260] Key type .llcrypt unregistered [ 2330.109698] Key type ._llcrypt unregistered [ 2356.199323] alg: No test for adler32 (adler32-zlib) [ 2356.955635] Key type ._llcrypt registered [ 2356.957925] Key type .llcrypt registered [ 2357.146307] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 2358.429738] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2359.047921] Lustre: Lustre: Build Version: 2.15.8_1_g00e7d23 [ 2359.272846] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 2359.289796] LNet: Accept secure, port 988 [ 2360.984164] Key type lgssc registered [ 2362.744554] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2373.164529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2390.820124] LustreError: 84531:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.204.144@tcp: failed processing log, type 1: rc = -5 [ 2421.536329] LustreError: 84531:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.204.144@tcp: failed processing log, type 4: rc = -110 [ 2451.361984] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2456.620039] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 2465.118053] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2473.176932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2473.360460] Lustre: *** cfs_fail_loc=706, val=0*** [ 2474.366307] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2474.463785] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2474.575130] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:161 [ 2479.570128] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 2479.598444] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2484.725103] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2489.085900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2489.454712] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2489.529176] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:161 [ 2494.219502] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 2501.514746] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2506.578484] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2509.795343] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2509.804933] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2509.829667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2510.306893] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2510.314140] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2511.330596] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2511.340618] Lustre: Skipped 2 previous similar messages [ 2514.873805] Lustre: server umount lustre-OST0000 complete [ 2520.036057] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2520.043606] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2520.052586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2524.728078] Lustre: server umount lustre-MDT0000 complete [ 2526.690730] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2526.701936] LustreError: Skipped 1 previous similar message [ 2527.812445] LustreError: 84539:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776202698 with bad export cookie 17169990594995226521 [ 2527.814230] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2527.822818] LustreError: 84539:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2528.055199] Lustre: server umount lustre-MDT0001 complete [ 2537.771900] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 2540.325404] Key type lgssc unregistered [ 2540.618349] LNet: 88269:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2540.639739] LNet: Removed LNI 192.168.204.144@tcp [ 2541.414733] Key type .llcrypt unregistered [ 2541.421509] Key type ._llcrypt unregistered [ 2563.584565] alg: No test for adler32 (adler32-zlib) [ 2564.349052] Key type ._llcrypt registered [ 2564.356044] Key type .llcrypt registered [ 2564.552798] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 2565.962966] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2566.496578] Lustre: Lustre: Build Version: 2.15.8_1_g00e7d23 [ 2566.769696] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 2566.775116] LNet: Accept secure, port 988 [ 2568.465922] Key type lgssc registered [ 2569.427425] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2577.228705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2582.547466] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 2590.811650] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2591.249244] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2591.347396] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2595.991383] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 2596.839220] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2596.867985] LustreError: Skipped 2 previous similar messages [ 2601.955423] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2601.967577] LustreError: Skipped 1 previous similar message [ 2605.380210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2605.811498] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2605.824917] LustreError: Skipped 2 previous similar messages [ 2605.855539] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2606.901032] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:193 [ 2611.045095] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 2612.214362] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2612.231918] LustreError: Skipped 5 previous similar messages [ 2620.484535] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2620.513757] LustreError: Skipped 5 previous similar messages [ 2620.816667] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2620.978790] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:193 [ 2625.343511] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 2641.387233] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2641.395305] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2641.401358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2641.418352] Lustre: Skipped 1 previous similar message [ 2644.852364] Lustre: server umount lustre-OST0000 complete [ 2651.617881] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2651.624201] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2651.644653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2651.653996] Lustre: Skipped 2 previous similar messages [ 2656.738416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2656.750636] Lustre: Skipped 1 previous similar message [ 2662.885602] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2662.896639] Lustre: Skipped 3 previous similar messages [ 2663.392929] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2663.491553] Lustre: server umount lustre-MDT0000 complete [ 2667.702952] Lustre: server umount lustre-MDT0001 complete [ 2678.240320] Lustre: 93143:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776202842/real 1776202842] req@00000000e173f816 x1862483561167360/t0(0) o251->MGC192.168.204.144@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776202848 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2678.295724] Lustre: server umount MGS complete [ 2689.333272] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 17:40:58 (1776202858) [ 2718.116620] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 2727.417407] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 2737.152929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2744.025734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2750.046754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2756.399531] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2762.311230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2762.374331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2766.629988] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 2775.963926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2776.053057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2776.230546] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2776.276253] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 2776.380814] Lustre: lustre-MDT0001: new disk, initializing [ 2776.502470] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2776.550311] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2776.567297] LustreError: 97844:0:(lod_dev.c:424:lod_sub_recovery_thread()) lustre-MDT0001-osd get update log failed: rc = -115 [ 2781.767806] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 2793.219738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2793.318618] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2793.333683] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2793.640813] Lustre: lustre-OST0000: new disk, initializing [ 2793.644910] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2793.711308] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2798.966088] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 2803.314334] Lustre: 99075:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@00000000939daddb x1862483561172224/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1776202986 ref 2 fl Rpc:RQU/2/0 rc 0/-115 job:'' [ 2809.514873] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2816.165201] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2816.213978] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2816.344428] Lustre: lustre-MDT0000: new disk, initializing [ 2816.501790] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2816.528508] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2817.568903] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2817.585925] Lustre: 99697:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@00000000cff3e6ff x1862483561175808/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1776203000 ref 2 fl Rpc:RQU/2/0 rc 0/-115 job:'' [ 2821.021552] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 2826.669681] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2826.687917] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2830.825079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2830.842312] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2851.305818] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2851.315506] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2851.347605] Lustre: Skipped 1 previous similar message [ 2851.355766] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2857.295554] Lustre: server umount lustre-OST0000 complete [ 2862.050994] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2862.051579] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2862.069811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2862.086169] Lustre: Skipped 2 previous similar messages [ 2862.100960] Lustre: Skipped 4 previous similar messages [ 2876.384230] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2876.528028] Lustre: server umount lustre-MDT0000 complete [ 2877.412024] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2877.435475] LustreError: Skipped 1 previous similar message [ 2896.352585] Lustre: 100890:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776203061/real 1776203061] req@000000007c69b4a0 x1862483561186752/t0(0) o251->MGC192.168.204.144@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776203067 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2896.430619] Lustre: server umount MGS complete [ 2896.435459] Lustre: Skipped 1 previous similar message [ 2908.254535] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 17:44:37 (1776203077) [ 2929.332880] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 2933.108669] Key type lgssc unregistered [ 2933.536046] LNet: 102310:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2933.545347] LNet: Removed LNI 192.168.204.144@tcp [ 2934.608531] Key type .llcrypt unregistered [ 2934.611766] Key type ._llcrypt unregistered [ 2955.338686] alg: No test for adler32 (adler32-zlib) [ 2956.099077] Key type ._llcrypt registered [ 2956.103906] Key type .llcrypt registered [ 2956.253885] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 2957.667527] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2958.226387] Lustre: Lustre: Build Version: 2.15.8_1_g00e7d23 [ 2958.476569] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 2958.490294] LNet: Accept secure, port 988 [ 2960.200145] Key type lgssc registered [ 2961.589682] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2971.479644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2973.054652] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2973.161573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2978.273940] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2978.490281] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 2983.399783] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2988.347449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2988.525406] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2988.863685] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2994.072537] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 3003.093089] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3009.580456] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3017.099158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3017.559062] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3022.394582] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 3023.741835] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 3030.670780] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3039.203272] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3039.206748] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3039.217458] LustreError: Skipped 1 previous similar message [ 3039.219524] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3039.259235] Lustre: Skipped 1 previous similar message [ 3044.321918] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3044.328742] Lustre: Skipped 1 previous similar message [ 3044.652227] Lustre: server umount lustre-OST0000 complete [ 3049.443225] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3049.449140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3049.473364] Lustre: Skipped 1 previous similar message [ 3049.475651] Lustre: Skipped 1 previous similar message [ 3054.562222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3054.585458] Lustre: Skipped 2 previous similar messages [ 3054.911851] LustreError: 106454:0:(ldlm_resource.c:1127:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (000000006de27b45) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 3056.942273] Lustre: server umount lustre-MDT0000 complete [ 3059.687991] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3059.714726] LustreError: Skipped 1 previous similar message [ 3062.242312] LustreError: 106456:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776203232 with bad export cookie 15509019862430267799 [ 3062.247051] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3062.258086] LustreError: 106456:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3062.773144] Lustre: server umount lustre-MDT0001 complete [ 3077.162970] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 3080.710786] Key type lgssc unregistered [ 3081.112786] LNet: 107262:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3081.127567] LNet: Removed LNI 192.168.204.144@tcp [ 3081.914615] Key type .llcrypt unregistered [ 3081.920146] Key type ._llcrypt unregistered [ 3096.078471] alg: No test for adler32 (adler32-zlib) [ 3096.838544] Key type ._llcrypt registered [ 3096.843127] Key type .llcrypt registered [ 3096.942945] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 3097.720288] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3098.118901] Lustre: Lustre: Build Version: 2.15.8_1_g00e7d23 [ 3098.346118] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 3098.351410] LNet: Accept secure, port 988 [ 3100.050205] Key type lgssc registered [ 3101.336224] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3108.326878] LustreError: 108383:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 716 sleeping [ 3108.346645] LustreError: 108387:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 716 waking [ 3108.351462] LustreError: 108383:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 716 awake: rc=4982 [ 3108.355105] LustreError: 108383:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3108.368692] LustreError: 108383:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3108.372136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3110.046416] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3110.151215] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3115.489256] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3119.242912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3119.498616] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3125.546559] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 3131.472494] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing lsmod [ 3134.868883] LustreError: 109688:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 716 sleeping [ 3134.913485] LustreError: 109696:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 716 waking [ 3134.918959] LustreError: 109688:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 716 awake: rc=4956 [ 3134.928147] LustreError: 109688:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3134.941155] LustreError: 109688:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3134.961572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3135.488937] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3140.066331] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3140.072861] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3140.096750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3140.592203] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3140.598054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3140.623190] Lustre: Skipped 1 previous similar message [ 3140.640981] Lustre: Skipped 1 previous similar message [ 3145.700158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3145.718905] Lustre: Skipped 2 previous similar messages [ 3149.806437] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3149.814730] Lustre: Skipped 2 previous similar messages [ 3153.376895] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3153.443833] Lustre: server umount lustre-MDT0000 complete [ 3154.915189] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3154.927084] LustreError: Skipped 2 previous similar messages [ 3157.663514] LustreError: 108400:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776203328 with bad export cookie 4235743870537944582 [ 3157.674500] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3157.678730] LustreError: 108400:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3158.052424] Lustre: server umount lustre-MDT0001 complete [ 3169.248157] Lustre: 110288:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776203333/real 1776203333] req@00000000529947e9 x1862484119006144/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776203339 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3169.515280] Lustre: server umount lustre-OST0000 complete [ 3180.736603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3181.301877] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3181.432889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3186.548320] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 3186.658363] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3196.109658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3196.496942] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3201.172557] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 3211.582401] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3218.895499] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3227.125485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3227.554631] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3232.101150] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 3232.743644] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:65 [ 3240.515596] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3252.712546] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3252.723139] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3252.742231] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3257.325869] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3257.347364] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3258.457694] Lustre: server umount lustre-OST0000 complete [ 3262.953270] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3262.967357] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3262.985084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3262.997223] Lustre: Skipped 2 previous similar messages [ 3268.886995] Lustre: server umount lustre-MDT0000 complete [ 3272.676094] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3272.709291] LustreError: Skipped 2 previous similar messages [ 3273.483596] LustreError: 110744:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776203444 with bad export cookie 4235743870537945282 [ 3273.493156] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3273.493165] LustreError: 110744:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3291.793361] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 3295.799424] Key type lgssc unregistered [ 3296.309640] LNet: 114422:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3296.324969] LNet: Removed LNI 192.168.204.144@tcp [ 3297.761697] Key type .llcrypt unregistered [ 3297.763119] Key type ._llcrypt unregistered [ 3323.795643] alg: No test for adler32 (adler32-zlib) [ 3324.563890] Key type ._llcrypt registered [ 3324.567297] Key type .llcrypt registered [ 3324.694248] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 3325.947440] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3326.622960] Lustre: Lustre: Build Version: 2.15.8_1_g00e7d23 [ 3326.880777] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 3326.885857] LNet: Accept secure, port 988 [ 3328.570104] Key type lgssc registered [ 3330.332635] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3341.857844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3343.682618] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3343.759896] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3348.963578] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3349.876707] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 3354.085449] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3359.202617] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3359.629779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3359.961373] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3364.943535] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 3374.534037] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3381.423236] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3388.792776] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3389.210138] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3393.341383] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 3395.096913] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 3403.625343] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3408.771154] Lustre: Setting parameter lustre-client.llite.some_wrong_param in log lustre-client [ 3413.058568] Lustre: Modifying parameter lustre-client.llite.some_wrong_param in log lustre-client [ 3419.108369] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3419.118940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3419.140379] Lustre: Skipped 1 previous similar message [ 3419.156436] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3424.235089] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3424.247206] Lustre: Skipped 2 previous similar messages [ 3429.359566] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3429.364521] Lustre: Skipped 1 previous similar message [ 3431.392121] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3431.635855] Lustre: server umount lustre-OST0000 complete [ 3437.025651] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3437.049565] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3437.068551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3439.586781] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3444.707139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3444.720732] Lustre: Skipped 3 previous similar messages [ 3450.337154] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3450.478079] Lustre: server umount lustre-MDT0000 complete [ 3454.154092] LustreError: 115879:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776203624 with bad export cookie 7441009148888991197 [ 3454.157260] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3454.169984] LustreError: 115879:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3454.512099] Lustre: server umount lustre-MDT0001 complete [ 3466.237846] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 3469.193188] Key type lgssc unregistered [ 3469.507915] LNet: 119662:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3469.532482] LNet: Removed LNI 192.168.204.144@tcp [ 3470.435645] Key type .llcrypt unregistered [ 3470.440537] Key type ._llcrypt unregistered [ 3488.540903] alg: No test for adler32 (adler32-zlib) [ 3489.304110] Key type ._llcrypt registered [ 3489.309394] Key type .llcrypt registered [ 3489.471551] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 3490.794229] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3491.912618] Lustre: Lustre: Build Version: 2.15.8_1_g00e7d23 [ 3492.298753] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 3492.306143] LNet: Accept secure, port 988 [ 3494.080159] Key type lgssc registered [ 3495.513438] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3505.338964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3506.882516] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3506.947517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3512.293979] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3512.809139] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 3517.411604] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3522.535350] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3523.178309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3523.833108] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3529.128892] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 3538.492076] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3545.690293] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3554.953483] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3555.481840] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3557.563417] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 3561.280480] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 3571.883955] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3588.584864] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3588.602257] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3588.631722] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3589.607475] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3589.610076] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3589.631519] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3589.660263] Lustre: Skipped 1 previous similar message [ 3592.043287] Lustre: server umount lustre-OST0000 complete [ 3599.851879] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3599.882072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3602.633836] Lustre: server umount lustre-MDT0000 complete [ 3604.971287] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3604.992599] LustreError: Skipped 1 previous similar message [ 3606.989924] LustreError: 120934:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776203777 with bad export cookie 906092757429592143 [ 3607.001231] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3607.003645] LustreError: 120934:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3607.503989] Lustre: server umount lustre-MDT0001 complete [ 3621.363556] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 3624.710180] Key type lgssc unregistered [ 3625.015414] LNet: 124633:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3625.032155] LNet: Removed LNI 192.168.204.144@tcp [ 3625.885141] Key type .llcrypt unregistered [ 3625.887758] Key type ._llcrypt unregistered [ 3652.138682] alg: No test for adler32 (adler32-zlib) [ 3652.906977] Key type ._llcrypt registered [ 3652.909497] Key type .llcrypt registered [ 3653.055985] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 3654.283171] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3655.192244] Lustre: Lustre: Build Version: 2.15.8_1_g00e7d23 [ 3655.396940] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 3655.407618] LNet: Accept secure, port 988 [ 3657.176128] Key type lgssc registered [ 3658.366159] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3665.996194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3667.352466] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3667.410941] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3671.699874] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 3672.545393] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3677.665501] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3679.692922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3679.968357] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3684.213626] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 3692.098900] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3698.236167] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3704.877297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3705.211462] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3711.224122] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 3717.497518] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 3721.688340] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3728.039307] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash in log lustre-MDT0000 [ 3732.088149] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids in log lustre-MDT0000 [ 3732.092145] Lustre: Skipped 2 previous similar messages [ 3737.009503] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash in log lustre-MDT0000 [ 3737.014448] Lustre: Skipped 2 previous similar messages [ 3737.697838] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3737.725618] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3743.389736] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids in log lustre-MDT0000 [ 3743.393815] Lustre: Skipped 2 previous similar messages [ 3744.082388] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.44@tcp 192.168.0.[2,10]@tcp [ 3750.889089] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3750.900594] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3750.911917] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3751.395794] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3751.412695] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3751.435424] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3753.463087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3753.475214] Lustre: Skipped 1 previous similar message [ 3756.825605] Lustre: server umount lustre-OST0000 complete [ 3762.144968] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3762.158397] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3762.189070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3767.217315] Lustre: server umount lustre-MDT0000 complete [ 3768.803337] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3768.820607] LustreError: Skipped 1 previous similar message [ 3771.442066] LustreError: 129428:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776203942 with bad export cookie 11439984017997294916 [ 3771.451617] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3771.467342] LustreError: 129428:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3772.016132] Lustre: server umount lustre-MDT0001 complete [ 3785.434612] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 3788.482262] Key type lgssc unregistered [ 3788.715844] LNet: 130591:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3788.725669] LNet: Removed LNI 192.168.204.144@tcp [ 3789.565943] Key type .llcrypt unregistered [ 3789.572184] Key type ._llcrypt unregistered [ 3819.625039] alg: No test for adler32 (adler32-zlib) [ 3820.380692] Key type ._llcrypt registered [ 3820.383777] Key type .llcrypt registered [ 3820.479625] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 3821.444568] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3822.070461] Lustre: Lustre: Build Version: 2.15.8_1_g00e7d23 [ 3822.340562] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 3822.346813] LNet: Accept secure, port 988 [ 3824.080186] Key type lgssc registered [ 3825.297545] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3833.478558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3835.125417] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3835.194718] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3835.237358] Lustre: lustre-MDT0000: nosquash_nids set to 2@gni 0@lo 192.168.204.44@tcp 192.168.0.[2,10]@tcp [ 3835.267411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3839.484883] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 3840.488028] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3845.605209] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3849.352623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3849.598896] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3849.623164] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.44@tcp 192.168.0.[2,10]@tcp [ 3849.684322] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3854.341321] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 3863.046500] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3869.545838] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3877.181170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3877.714628] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3883.187093] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 3885.941973] Lustre: lustre-OST0000: deleting orphan objects from 0x0:167 to 0x0:193 [ 3892.170708] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3901.409355] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3901.426139] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3901.444194] Lustre: Skipped 1 previous similar message [ 3901.456982] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3901.472731] Lustre: Skipped 1 previous similar message [ 3906.531958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3906.545681] Lustre: Skipped 1 previous similar message [ 3906.974940] Lustre: server umount lustre-OST0000 complete [ 3916.259407] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3916.273749] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3916.291928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3918.370954] Lustre: server umount lustre-MDT0000 complete [ 3921.889912] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3921.904650] LustreError: Skipped 1 previous similar message [ 3923.046274] LustreError: 132203:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776204093 with bad export cookie 10829424570827248534 [ 3923.055882] LustreError: 132203:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3923.056247] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3923.501903] Lustre: server umount lustre-MDT0001 complete [ 3937.779762] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 3941.613703] Key type lgssc unregistered [ 3942.004369] LNet: 135944:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3942.017930] LNet: Removed LNI 192.168.204.144@tcp [ 3943.176682] Key type .llcrypt unregistered [ 3943.182929] Key type ._llcrypt unregistered [ 3976.308660] alg: No test for adler32 (adler32-zlib) [ 3977.067720] Key type ._llcrypt registered [ 3977.075549] Key type .llcrypt registered [ 3977.296553] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 3990.425739] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 3991.208317] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3992.044488] Lustre: Lustre: Build Version: 2.15.8_1_g00e7d23 [ 3992.327784] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 3992.332962] LNet: Accept secure, port 988 [ 3994.032124] Key type lgssc registered [ 3995.105714] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4005.202911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4012.825537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4020.086620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4026.936567] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4034.784752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4034.934210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4036.248893] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4036.325828] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4036.457314] Lustre: lustre-MDT0000: new disk, initializing [ 4036.593502] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4036.627665] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4042.248931] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 4053.215947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4053.273828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4053.382723] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4053.431153] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 4053.437891] Lustre: Skipped 1 previous similar message [ 4053.545474] Lustre: lustre-MDT0001: new disk, initializing [ 4053.643569] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4053.690217] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4053.709098] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4058.597177] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 4068.750844] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4075.812602] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4083.552362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4083.611140] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4083.620778] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4083.881545] Lustre: lustre-OST0000: new disk, initializing [ 4083.886215] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 4083.954645] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4088.416964] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 4091.497588] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4091.515437] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4098.580635] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4103.929242] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 4104.172963] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4109.456783] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 4109.741962] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4112.353383] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4112.358827] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4112.362238] LustreError: Skipped 1 previous similar message [ 4112.366741] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4112.388043] Lustre: Skipped 1 previous similar message [ 4112.416245] Lustre: Skipped 1 previous similar message [ 4117.477725] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4117.488143] Lustre: Skipped 1 previous similar message [ 4117.934545] Lustre: server umount lustre-OST0000 complete [ 4122.596275] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4122.600883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4122.616687] Lustre: Skipped 1 previous similar message [ 4122.627515] Lustre: Skipped 1 previous similar message [ 4127.720343] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4127.725662] Lustre: Skipped 1 previous similar message [ 4128.600904] Lustre: server umount lustre-MDT0000 complete [ 4132.809948] LustreError: 139477:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776204303 with bad export cookie 14775364905116212181 [ 4132.812610] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4132.833940] LustreError: 139477:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4133.253627] Lustre: server umount lustre-MDT0001 complete [ 4142.912598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4143.514919] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4143.586280] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4148.345578] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 4148.717832] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4153.835326] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4157.748423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4158.198051] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4163.255655] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 4171.527393] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4179.418304] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4187.739537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4188.051287] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4194.012604] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 4204.060423] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4210.030902] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 4210.292886] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4215.181348] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 4215.489310] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4224.940492] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4225.002553] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4225.007546] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 4225.105888] Lustre: lustre-OST0001: new disk, initializing [ 4225.109449] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 4225.194336] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4229.875748] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 4230.740074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4230.751120] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4239.617333] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 4239.859446] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4245.217468] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 4245.517687] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4252.537696] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9f1c88ce3000.ost_server_uuid 40 [ 4254.410444] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9f1c88ce3000.ost_server_uuid in IDLE state after 0 sec [ 4266.622703] Lustre: server umount lustre-OST0001 complete [ 4266.984881] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4266.990928] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4267.002263] LustreError: Skipped 1 previous similar message [ 4267.006389] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4267.014403] Lustre: Skipped 1 previous similar message [ 4267.035326] LustreError: Skipped 1 previous similar message [ 4271.075120] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4271.088340] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4271.102147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4272.101235] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4272.124630] LustreError: Skipped 1 previous similar message [ 4276.792792] Lustre: server umount lustre-OST0000 complete [ 4281.325826] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4281.334287] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4281.348772] Lustre: Skipped 1 previous similar message [ 4281.359049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4281.367847] Lustre: Skipped 2 previous similar messages [ 4287.425463] Lustre: server umount lustre-MDT0000 complete [ 4287.477023] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4291.948692] LustreError: 143824:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776204462 with bad export cookie 14775364905116213035 [ 4291.955974] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4291.964675] LustreError: 143824:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4314.238399] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 4317.008372] Key type lgssc unregistered [ 4317.331603] LNet: 150501:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4317.346098] LNet: Removed LNI 192.168.204.144@tcp [ 4318.315430] Key type .llcrypt unregistered [ 4318.320644] Key type ._llcrypt unregistered [ 4355.134883] alg: No test for adler32 (adler32-zlib) [ 4355.892426] Key type ._llcrypt registered [ 4355.895158] Key type .llcrypt registered [ 4356.006250] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 4372.978845] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 4374.177281] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4375.079813] Lustre: Lustre: Build Version: 2.15.8_1_g00e7d23 [ 4375.422748] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 4375.430626] LNet: Accept secure, port 988 [ 4377.144136] Key type lgssc registered [ 4379.077096] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4389.681152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4396.173535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4403.126852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4409.668913] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4416.005341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4416.066240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4417.319988] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4417.358635] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4417.451598] Lustre: lustre-MDT0000: new disk, initializing [ 4417.534258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4417.584960] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4422.220926] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 4433.367894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4433.480154] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4433.653969] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4433.774510] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 4433.792451] Lustre: Skipped 1 previous similar message [ 4433.959772] Lustre: lustre-MDT0001: new disk, initializing [ 4434.114838] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4434.174265] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4434.192993] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4440.463585] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 4452.886863] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4460.125228] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4467.600136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4467.707201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4467.716840] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4467.952951] Lustre: lustre-OST0000: new disk, initializing [ 4467.967252] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 4468.089435] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4473.417815] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 4475.453380] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4475.463864] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4482.925798] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4488.252561] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 4488.534186] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4494.151932] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 4494.504521] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4501.480054] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4501.487246] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4501.541153] Lustre: Skipped 1 previous similar message [ 4506.609494] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4506.633206] Lustre: Skipped 2 previous similar messages [ 4511.200202] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 4511.421075] Lustre: server umount lustre-OST0000 complete [ 4516.321517] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4516.333567] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4516.354985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4521.684411] Lustre: server umount lustre-MDT0000 complete [ 4521.972852] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4522.002629] LustreError: Skipped 1 previous similar message [ 4526.513136] LustreError: 154613:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776204697 with bad export cookie 4430396755632344011 [ 4526.518724] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4526.527033] LustreError: 154613:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4526.982436] Lustre: server umount lustre-MDT0001 complete [ 4537.153437] Lustre: DEBUG MARKER: == conf-sanity test 47: server restart does not make client loss lru_resize settings ========================================================== 18:11:45 (1776204705) [ 4555.207377] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 4566.431740] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 4578.129056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4585.178900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4592.057764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4600.136618] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4608.360098] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4608.455057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4608.778910] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4608.816522] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4608.962195] Lustre: lustre-MDT0000: new disk, initializing [ 4609.064149] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4609.079638] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4614.455508] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 4625.578750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4625.666279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4625.740976] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4625.779855] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 4625.785177] Lustre: Skipped 1 previous similar message [ 4625.864835] Lustre: lustre-MDT0001: new disk, initializing [ 4625.987176] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4626.028316] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4626.043748] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4631.220836] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 4642.495621] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4649.251110] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4657.276287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4657.353750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4657.364180] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4657.721633] Lustre: lustre-OST0000: new disk, initializing [ 4657.729208] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 4657.874929] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4663.187661] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 4663.934115] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4663.954950] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4675.050314] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4679.850047] Lustre: Failing over lustre-OST0000 [ 4681.702692] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4681.709680] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4681.720524] Lustre: Skipped 1 previous similar message [ 4681.726076] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4681.736409] Lustre: Skipped 1 previous similar message [ 4682.040913] Lustre: server umount lustre-OST0000 complete [ 4692.000164] Lustre: 152838:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776204855/real 1776204855] req@000000009def2fd5 x1862485458064768/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1776204862 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 4692.039555] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4696.531044] Lustre: 152839:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776204860/real 1776204860] req@00000000f8caf2b3 x1862485458065664/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1776204867 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4702.276616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4702.715352] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4702.734440] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4705.653831] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4707.595665] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 4711.871304] Lustre: Failing over lustre-MDT0000 [ 4711.907878] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4711.908354] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4711.921246] Lustre: Skipped 1 previous similar message [ 4711.938150] Lustre: Skipped 2 previous similar messages [ 4712.054154] Lustre: server umount lustre-MDT0000 complete [ 4713.958330] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4713.971708] LustreError: Skipped 2 previous similar messages [ 4715.921984] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4719.078642] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4719.089222] LustreError: Skipped 2 previous similar messages [ 4721.056151] Lustre: 152839:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776204884/real 1776204884] req@0000000047dae3b6 x1862485458071936/t0(0) o400->MGC192.168.204.144@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776204891 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4721.088688] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4723.080759] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4723.109225] LustreError: Skipped 4 previous similar messages [ 4732.387637] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4732.413229] LustreError: Skipped 10 previous similar messages [ 4733.225992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4738.534481] Lustre: Evicted from MGS (at 192.168.204.144@tcp) after server handle changed from 0x3d7bee6041824f21 to 0x3d7bee60418253ea [ 4738.555614] Lustre: MGC192.168.204.144@tcp: Connection restored to 192.168.204.144@tcp (at 0@lo) [ 4738.768466] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 4738.850982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4738.932707] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4739.442848] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4744.171385] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 4744.191364] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4744.288258] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 4744.311342] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 4754.401830] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4754.403128] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4754.415889] LustreError: Skipped 1 previous similar message [ 4754.419707] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4754.427245] Lustre: Skipped 2 previous similar messages [ 4754.443229] Lustre: Skipped 1 previous similar message [ 4759.362402] Lustre: server umount lustre-OST0000 complete [ 4764.664128] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4770.787687] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4770.813371] LustreError: Skipped 8 previous similar messages [ 4775.333215] LustreError: 161609:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776204946 with bad export cookie 4430396755632346090 [ 4775.347522] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4775.350785] LustreError: 161609:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4775.789669] Lustre: server umount lustre-MDT0001 complete [ 4775.791446] Lustre: Skipped 1 previous similar message [ 4791.755695] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 4795.575087] Key type lgssc unregistered [ 4796.008851] LNet: 167477:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4796.024569] LNet: Removed LNI 192.168.204.144@tcp [ 4797.275991] Key type .llcrypt unregistered [ 4797.280944] Key type ._llcrypt unregistered [ 4828.391521] alg: No test for adler32 (adler32-zlib) [ 4829.146800] Key type ._llcrypt registered [ 4829.148861] Key type .llcrypt registered [ 4829.327750] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 4831.220414] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4831.746499] Lustre: Lustre: Build Version: 2.15.8_1_g00e7d23 [ 4832.048454] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 4832.052704] LNet: Accept secure, port 988 [ 4833.793547] Key type lgssc registered [ 4834.926246] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4845.286900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4847.040249] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4847.110115] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4852.201156] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4853.102357] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 4857.313172] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4861.412074] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4863.981622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4864.405647] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4870.063856] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 4880.064785] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4887.680491] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4896.239367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4896.814814] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4897.850875] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:65 [ 4901.695839] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 4910.338794] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4944.261641] Lustre: lustre-MDT0000: Client eaf13efc-038b-4f94-94dc-ad67c908abdb (at 192.168.204.44@tcp) reconnecting [ 4955.054503] Lustre: lustre-MDT0000: Client eaf13efc-038b-4f94-94dc-ad67c908abdb (at 192.168.204.44@tcp) reconnecting [ 5047.206625] Lustre: lustre-MDT0000: Client eaf13efc-038b-4f94-94dc-ad67c908abdb (at 192.168.204.44@tcp) reconnecting [ 5152.693143] Lustre: lustre-MDT0000: Client eaf13efc-038b-4f94-94dc-ad67c908abdb (at 192.168.204.44@tcp) reconnecting [ 5320.633221] Lustre: lustre-MDT0000: Client eaf13efc-038b-4f94-94dc-ad67c908abdb (at 192.168.204.44@tcp) reconnecting [ 5386.123814] Lustre: lustre-MDT0000: Client eaf13efc-038b-4f94-94dc-ad67c908abdb (at 192.168.204.44@tcp) reconnecting [ 5459.932164] Lustre: lustre-MDT0000: Client eaf13efc-038b-4f94-94dc-ad67c908abdb (at 192.168.204.44@tcp) reconnecting [ 5482.634864] Lustre: Failing over lustre-MDT0000 [ 5482.910772] Lustre: server umount lustre-MDT0000 complete [ 5483.508768] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5483.520292] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5483.534482] Lustre: Skipped 2 previous similar messages [ 5483.568515] LustreError: Skipped 2 previous similar messages [ 5485.443532] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5488.619588] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5488.634836] LustreError: Skipped 2 previous similar messages [ 5490.656956] Lustre: 168273:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776205654/real 1776205654] req@00000000bc7bce23 x1862485936568000/t0(0) o400->MGC192.168.204.144@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776205661 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 5490.682273] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5490.706478] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5490.728455] LustreError: Skipped 3 previous similar messages [ 5495.785912] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5495.807308] LustreError: Skipped 3 previous similar messages [ 5504.484241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5507.045312] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5507.059471] LustreError: Skipped 11 previous similar messages [ 5508.081466] Lustre: Evicted from MGS (at 192.168.204.144@tcp) after server handle changed from 0x1fa73b23940c3b24 to 0x1fa73b23940ea745 [ 5508.104444] Lustre: MGC192.168.204.144@tcp: Connection restored to 192.168.204.144@tcp (at 0@lo) [ 5508.472416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5508.549989] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5508.977598] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5513.722293] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 5513.781640] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 5513.872766] Lustre: lustre-OST0000: deleting orphan objects from 0x0:68 to 0x0:97 [ 5514.455775] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 5525.579426] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5527.675044] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5549.542097] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5549.551606] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5549.566134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5549.574969] Lustre: Skipped 2 previous similar messages [ 5549.597630] Lustre: Skipped 1 previous similar message [ 5554.658476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5554.666558] Lustre: Skipped 3 previous similar messages [ 5559.779976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5561.312142] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 5561.512953] Lustre: server umount lustre-MDT0000 complete [ 5564.912723] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5564.932982] LustreError: Skipped 6 previous similar messages [ 5566.700164] LustreError: 168942:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776205737 with bad export cookie 2280856760280328005 [ 5566.703412] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5566.710097] LustreError: 168942:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5570.018733] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5570.033110] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5570.036945] Lustre: Skipped 2 previous similar messages [ 5578.209808] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5578.228865] Lustre: Skipped 2 previous similar messages [ 5581.281618] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 5581.542404] Lustre: server umount lustre-MDT0001 complete [ 5592.500636] Lustre: server umount lustre-OST0000 complete [ 5605.987873] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 5616.441840] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 5629.030638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5637.031337] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5644.702918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5652.318792] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5659.563604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5659.660787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5659.937569] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5659.977329] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5660.054618] Lustre: lustre-MDT0000: new disk, initializing [ 5660.137963] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5660.156965] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5664.932513] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 5677.589373] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5677.651733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5677.736645] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5677.774652] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 5677.780684] Lustre: Skipped 1 previous similar message [ 5677.883926] Lustre: lustre-MDT0001: new disk, initializing [ 5678.004358] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5678.064081] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5678.082844] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5683.123328] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 5694.302306] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5702.400263] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5710.864304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5710.980786] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5710.988974] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 5711.307589] Lustre: lustre-OST0000: new disk, initializing [ 5711.322340] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5711.431231] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5716.880520] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 5719.621968] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5719.632722] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5728.075931] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5733.435402] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 5733.653831] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5739.085605] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 5739.341890] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5745.639671] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5745.649956] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5745.652537] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5745.694492] Lustre: Skipped 1 previous similar message [ 5747.683830] Lustre: server umount lustre-OST0000 complete [ 5754.852361] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5754.863705] LustreError: Skipped 1 previous similar message [ 5754.870262] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5758.892032] Lustre: server umount lustre-MDT0000 complete [ 5760.993214] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5761.019063] LustreError: Skipped 5 previous similar messages [ 5764.883859] LustreError: 175902:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776205935 with bad export cookie 2280856760280328726 [ 5764.903049] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5764.912671] LustreError: 175902:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5777.895372] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 18:32:26 (1776205946) [ 5797.207102] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 5809.117583] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 5821.741228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5830.077771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5836.788810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5844.015656] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5851.869855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5851.988559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5852.367181] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5852.419820] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5852.528815] Lustre: lustre-MDT0000: new disk, initializing [ 5852.667843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5852.713725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5857.677951] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 5869.469436] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5869.532400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5869.602178] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5869.640953] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 5869.645227] Lustre: Skipped 1 previous similar message [ 5869.767436] Lustre: lustre-MDT0001: new disk, initializing [ 5869.963948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5869.981847] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5874.425315] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 5886.373028] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5894.122837] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5901.920914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5902.014677] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5902.032311] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 5902.357437] Lustre: lustre-OST0000: new disk, initializing [ 5902.365867] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5907.009437] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 5910.084460] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5910.090796] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5916.827514] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5925.866144] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5925.871564] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5925.872325] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5925.878039] Lustre: Skipped 2 previous similar messages [ 5925.923778] Lustre: Skipped 5 previous similar messages [ 5931.276202] Lustre: server umount lustre-OST0000 complete [ 5931.281518] Lustre: Skipped 1 previous similar message [ 5946.338781] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5946.354483] LustreError: Skipped 3 previous similar messages [ 5948.302466] LustreError: 182897:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776206119 with bad export cookie 2280856760280329573 [ 5948.304631] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5948.315075] LustreError: 182897:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5958.635505] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 18:35:27 (1776206127) [ 5977.802249] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 5988.299295] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 6000.416542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6007.117030] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6013.223705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6019.725511] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6027.859880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6027.934473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6028.326033] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6028.369361] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 6028.530743] Lustre: lustre-MDT0000: new disk, initializing [ 6028.644571] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6028.652542] Lustre: Skipped 2 previous similar messages [ 6028.669201] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6033.963501] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 6045.571729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6045.690960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6045.847436] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6046.280980] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6052.677282] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 6066.119399] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6074.324548] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6081.823293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6081.910087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6081.918342] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 6082.089562] Lustre: lustre-OST0000: new disk, initializing [ 6082.093235] Lustre: Skipped 1 previous similar message [ 6082.096243] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 6082.102495] Lustre: Skipped 2 previous similar messages [ 6086.770618] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 6091.837160] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6091.854426] Lustre: Skipped 1 previous similar message [ 6091.858441] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6096.670782] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6107.625291] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6107.628084] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6107.637124] LustreError: Skipped 1 previous similar message [ 6107.637474] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6107.637479] Lustre: Skipped 4 previous similar messages [ 6107.686745] Lustre: Skipped 3 previous similar messages [ 6110.472466] Lustre: server umount lustre-OST0000 complete [ 6110.485633] Lustre: Skipped 2 previous similar messages [ 6117.857609] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6122.983107] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6122.990963] LustreError: Skipped 1 previous similar message [ 6125.815433] LustreError: 189362:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776206296 with bad export cookie 2280856760280330665 [ 6125.817434] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6125.826944] LustreError: 189362:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6140.830848] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 6143.893266] Key type lgssc unregistered [ 6144.244593] LNet: 193330:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6144.260731] LNet: Removed LNI 192.168.204.144@tcp [ 6145.127142] Key type .llcrypt unregistered [ 6145.130600] Key type ._llcrypt unregistered [ 6172.873787] alg: No test for adler32 (adler32-zlib) [ 6173.626575] Key type ._llcrypt registered [ 6173.629075] Key type .llcrypt registered [ 6173.769587] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 6175.128160] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6176.172735] Lustre: Lustre: Build Version: 2.15.8_1_g00e7d23 [ 6176.402257] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 6176.422270] LNet: Accept secure, port 988 [ 6178.240570] Key type lgssc registered [ 6180.297741] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6189.572978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6191.342664] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6191.449936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6196.724209] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6196.995547] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 6201.833887] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6206.948663] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6207.241593] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6207.651949] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6213.079240] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 6222.883676] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6230.477787] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6237.290958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6237.595952] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6241.722079] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 6241.786766] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 6250.207922] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6269.412239] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6269.419069] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6269.448057] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6272.482731] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6272.485332] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6272.502436] Lustre: Skipped 1 previous similar message [ 6274.881695] Lustre: server umount lustre-OST0000 complete [ 6279.652322] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6279.661383] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6279.676884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6282.722397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6282.724345] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6285.995261] Lustre: server umount lustre-MDT0000 complete [ 6287.841991] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6287.854169] LustreError: Skipped 1 previous similar message [ 6290.813645] LustreError: 194797:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776206461 with bad export cookie 4694355429147664081 [ 6290.818453] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6290.830383] LustreError: 194797:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6291.271585] Lustre: server umount lustre-MDT0001 complete [ 6306.582966] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 6309.363331] Key type lgssc unregistered [ 6309.557726] LNet: 198494:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6309.567801] LNet: Removed LNI 192.168.204.144@tcp [ 6310.369630] Key type .llcrypt unregistered [ 6310.371622] Key type ._llcrypt unregistered [ 6335.461309] alg: No test for adler32 (adler32-zlib) [ 6336.215711] Key type ._llcrypt registered [ 6336.220048] Key type .llcrypt registered [ 6336.419128] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 6338.073212] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6338.771347] Lustre: Lustre: Build Version: 2.15.8_1_g00e7d23 [ 6339.102871] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 6339.109211] LNet: Accept secure, port 988 [ 6340.896412] Key type lgssc registered [ 6342.260871] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6351.141725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6352.782993] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6352.829793] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6357.993412] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6358.544623] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 6363.109700] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6368.226662] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6369.692992] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6370.268096] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6375.842229] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 6385.064466] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6394.332363] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6403.043541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6403.420263] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6408.983689] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 6418.111663] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6427.962750] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 6436.323198] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6436.334167] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6436.357324] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6438.373767] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6438.381751] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6438.397620] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6439.832400] Lustre: lustre-OST0000: Not available for connect from 192.168.204.44@tcp (stopping) [ 6439.851774] Lustre: Skipped 1 previous similar message [ 6441.252472] Lustre: server umount lustre-OST0000 complete [ 6450.174115] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9f1c98652000.ost_server_uuid 40 [ 6452.463463] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6470.116576] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6470.132871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6470.140632] Lustre: Skipped 1 previous similar message [ 6473.967211] Lustre: server umount lustre-MDT0000 complete [ 6475.240947] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6475.264521] LustreError: Skipped 1 previous similar message [ 6478.261669] LustreError: 199960:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776206649 with bad export cookie 9076901905850437608 [ 6478.268637] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6478.277091] LustreError: 199960:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6478.642635] Lustre: server umount lustre-MDT0001 complete [ 6488.722972] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 18:44:17 (1776206657) [ 6495.545373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6496.146384] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6496.230097] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6501.208154] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 6510.612800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6510.968196] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6515.822369] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 6524.600592] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6531.156460] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6538.153720] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6538.555779] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6543.692211] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 6547.780910] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 6553.371573] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6560.648036] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6560.740044] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6560.756103] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 6561.002482] Lustre: lustre-OST0001: new disk, initializing [ 6561.010156] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 6561.201290] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6565.231867] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 6567.057265] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6567.068949] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6576.090444] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6582.163968] Lustre: lustre-OST0000: Not available for connect from 192.168.204.44@tcp (stopping) [ 6582.758014] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6582.773307] Lustre: Skipped 1 previous similar message [ 6585.554263] Lustre: server umount lustre-OST0000 complete [ 6587.245797] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.204.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6587.262355] LustreError: Skipped 2 previous similar messages [ 6593.799814] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 6594.034614] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6599.813685] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 6600.053385] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6618.597565] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6618.606931] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6618.617583] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6618.623031] Lustre: Skipped 2 previous similar messages [ 6618.640340] Lustre: Skipped 3 previous similar messages [ 6621.547993] Lustre: server umount lustre-OST0001 complete [ 6628.832983] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6628.839652] LustreError: Skipped 1 previous similar message [ 6628.841765] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6628.864661] Lustre: Skipped 1 previous similar message [ 6633.229431] Lustre: server umount lustre-MDT0000 complete [ 6633.953100] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6633.970726] LustreError: Skipped 19 previous similar messages [ 6637.469610] LustreError: 203971:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776206808 with bad export cookie 9076901905850438665 [ 6637.482929] LustreError: 203971:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6637.488728] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6644.782114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6650.051358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6654.359417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6659.168454] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6668.292304] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 18:47:16 (1776206836) [ 6676.756415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6677.127452] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6677.200367] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6677.584597] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6683.062262] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 6691.762868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6691.845907] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6691.892593] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6697.138411] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 6708.127747] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6715.598708] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6723.005716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6723.240513] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6728.552568] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 6731.887898] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:129 [ 6737.728937] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6745.044877] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6745.183170] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6745.414887] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6745.421020] Lustre: Skipped 2 previous similar messages [ 6750.928484] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3 to 0x0:33 [ 6751.104492] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 6759.613557] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6762.977118] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6762.989454] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6763.013317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6763.024697] Lustre: Skipped 2 previous similar messages [ 6768.705786] Lustre: server umount lustre-OST0000 complete [ 6768.712782] Lustre: Skipped 1 previous similar message [ 6770.568465] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.204.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6770.588970] LustreError: Skipped 2 previous similar messages [ 6787.041249] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6797.281744] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6797.287788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6797.303463] Lustre: Skipped 4 previous similar messages [ 6797.323600] Lustre: Skipped 8 previous similar messages [ 6803.513550] Lustre: server umount lustre-MDT0000 complete [ 6803.515661] Lustre: Skipped 1 previous similar message [ 6807.537655] LustreError: 210420:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776206978 with bad export cookie 9076901905850440058 [ 6807.542749] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6807.551445] LustreError: 210420:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6813.759779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6817.751929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6821.252805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6825.170272] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6832.170746] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 18:50:01 (1776207001) [ 6849.245778] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 6858.948669] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 6871.943104] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6880.404475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6886.266215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6891.881942] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6898.889081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6898.969942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6899.268434] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6899.310097] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 6899.388953] Lustre: lustre-MDT0000: new disk, initializing [ 6899.506841] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6899.548935] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6905.093801] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 6915.770622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6915.862942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6915.984341] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6916.036437] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 6916.041885] Lustre: Skipped 1 previous similar message [ 6916.156429] Lustre: lustre-MDT0001: new disk, initializing [ 6916.302832] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6916.316274] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6920.915693] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 6929.675900] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6935.896175] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6944.250211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6944.341269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6944.348124] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 6944.739680] Lustre: lustre-OST0000: new disk, initializing [ 6944.746310] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 6949.897661] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 6950.481140] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6950.499870] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6960.061610] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6966.189958] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 6966.494933] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6972.103155] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 6972.445680] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6976.497774] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6976.513752] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6976.530849] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6976.544422] Lustre: Skipped 2 previous similar messages [ 6980.602653] Lustre: server umount lustre-OST0000 complete [ 6980.605321] Lustre: Skipped 1 previous similar message [ 6991.843073] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6991.850610] LustreError: Skipped 15 previous similar messages [ 6993.928151] LustreError: 218661:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776207164 with bad export cookie 9076901905850441794 [ 6993.931725] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6993.941661] LustreError: 218661:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7002.203606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7007.455443] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 7015.396346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7019.900879] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 7027.653827] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7033.743211] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7042.027179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7049.642165] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 7060.718699] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7066.371815] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 7066.575797] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7070.552115] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 7070.715943] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7073.775563] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7073.794284] LustreError: Skipped 1 previous similar message [ 7086.645067] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 7092.913495] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 7122.226272] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7126.709216] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7126.740683] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7126.744594] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to (at 0@lo) [ 7126.754126] Lustre: lustre-OST0000: deleting orphan objects from 0x0:34 to 0x0:65 [ 7126.784964] Lustre: Skipped 1 previous similar message [ 7126.788688] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:65 [ 7127.387091] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 7135.003967] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7136.457341] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7142.381136] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7142.386127] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7142.395140] Lustre: Skipped 5 previous similar messages [ 7142.409300] LustreError: Skipped 1 previous similar message [ 7142.416242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7142.430824] Lustre: Skipped 5 previous similar messages [ 7146.321564] Lustre: server umount lustre-OST0000 complete [ 7146.324393] Lustre: Skipped 3 previous similar messages [ 7159.440168] LustreError: 223008:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776207330 with bad export cookie 9076901905850442648 [ 7159.453716] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7168.285866] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 18:55:37 (1776207337) [ 7175.400592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7176.001846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7176.006566] Lustre: Skipped 6 previous similar messages [ 7180.287399] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 7189.425725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7195.141957] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 7203.544343] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7209.686956] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7216.114981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7221.037548] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 7229.375482] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7235.047590] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 7235.350689] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7241.271270] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 7241.611198] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7249.029101] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7249.219255] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7249.227707] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 7249.357478] Lustre: lustre-OST0001: new disk, initializing [ 7249.366440] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 7255.246610] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 7267.021247] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7273.037172] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 7273.473179] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7280.425948] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 7280.784843] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7287.283303] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7292.390869] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7292.401543] LustreError: Skipped 13 previous similar messages [ 7297.095209] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 7297.399858] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7301.370666] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 7301.559953] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7332.725357] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7337.975616] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7337.983841] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to (at 0@lo) [ 7337.989944] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7338.026461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7338.033945] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7338.164288] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 7345.983699] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7347.386484] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7387.104229] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 7405.571600] LustreError: 229203:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776207576 with bad export cookie 9076901905850444062 [ 7405.579716] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7405.590040] LustreError: 229203:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 7412.221699] Lustre: server umount lustre-MDT0001 complete [ 7412.225386] Lustre: Skipped 6 previous similar messages [ 7419.450967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7423.637469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7427.918314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7431.863865] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7439.316275] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 19:00:08 (1776207608) [ 7445.731717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7446.108376] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7446.138972] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 7450.687170] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 7459.160076] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7459.420449] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7459.487951] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 7465.267250] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 7473.937262] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7480.359426] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7487.634077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7487.859817] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7492.968587] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 7501.325093] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7510.779688] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7510.843043] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7516.104737] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 7524.786216] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7526.706198] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7532.729097] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 7533.019613] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7538.417533] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 7538.669376] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7545.769839] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9f1c98357800.ost_server_uuid 40 [ 7548.083525] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9f1c98357800.ost_server_uuid in IDLE state after 0 sec [ 7549.356081] Lustre: Permanently deactivating lustre-OST0001 [ 7549.372050] Lustre: Setting parameter lustre-OST0001-osc.osc.active in log lustre-client [ 7554.431798] Lustre: Permanently reactivating lustre-OST0001 [ 7559.664139] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7559.670749] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7559.675509] LustreError: Skipped 3 previous similar messages [ 7559.695427] Lustre: Skipped 11 previous similar messages [ 7559.706795] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7559.709752] Lustre: Skipped 14 previous similar messages [ 7591.802576] LustreError: 238140:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776207762 with bad export cookie 9076901905850445336 [ 7591.810165] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7591.821962] LustreError: 238140:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7599.338345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7604.021412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7607.930611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7612.118241] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7619.732731] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 19:03:08 (1776207788) [ 7620.938532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7628.123904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7628.360681] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7628.392146] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 7628.400652] Lustre: Skipped 5 previous similar messages [ 7633.383198] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 7643.332093] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7643.503536] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7649.225176] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 7658.595052] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7665.306384] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7671.986782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7672.067478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7672.328741] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7672.352845] Lustre: Permanently deactivating lustre-OST0000 [ 7677.368491] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 7679.077228] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7686.902404] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7694.568066] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7694.710845] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7694.986074] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7695.003655] Lustre: Skipped 11 previous similar messages [ 7700.633652] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 7701.691423] Lustre: lustre-OST0001: deleting orphan objects from 0x0:67 to 0x0:97 [ 7710.418285] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7712.731341] Lustre: Permanently reactivating lustre-OST0000 [ 7712.766728] Lustre: Modifying parameter lustre-OST0000-osc.osc.active in log lustre-client [ 7712.788793] Lustre: Skipped 4 previous similar messages [ 7720.076450] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7720.090367] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7720.110887] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to (at 0@lo) [ 7720.128026] Lustre: Skipped 1 previous similar message [ 7720.192827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7720.231706] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 7740.384643] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 7766.054341] LustreError: 245805:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776207936 with bad export cookie 9076901905850446967 [ 7766.057285] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7766.079987] LustreError: 245805:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7780.425785] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 7783.162681] Key type lgssc unregistered [ 7783.517859] LNet: 249975:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7783.528052] LNet: Removed LNI 192.168.204.144@tcp [ 7784.433790] Key type .llcrypt unregistered [ 7784.438301] Key type ._llcrypt unregistered [ 7808.503023] alg: No test for adler32 (adler32-zlib) [ 7809.266804] Key type ._llcrypt registered [ 7809.279056] Key type .llcrypt registered [ 7809.493273] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 7811.616647] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7812.148291] Lustre: Lustre: Build Version: 2.15.8_1_g00e7d23 [ 7812.396377] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 7812.399832] LNet: Accept secure, port 988 [ 7814.160158] Key type lgssc registered [ 7815.547472] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7820.499562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7827.240691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7828.668842] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7828.707935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7832.797472] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 7834.089385] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7839.206058] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7842.077146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7842.165936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7842.257745] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7842.299194] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 7842.307729] Lustre: Permanently deactivating lustre-MDT0001 [ 7842.543096] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7844.127073] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7846.970478] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 7854.958291] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7860.701837] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7867.161027] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7867.506690] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7872.171186] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 7872.618349] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7878.116967] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7878.731540] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7879.163916] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:193 [ 7885.761868] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7886.048798] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7891.251384] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 7891.455031] Lustre: lustre-OST0001: deleting orphan objects from 0x0:67 to 0x0:129 [ 7891.472041] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:67 to 0x2c0000400:97 [ 7900.313795] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7903.791996] Lustre: Permanently reactivating lustre-MDT0001 [ 7903.809464] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active in log lustre-client [ 7903.814507] Lustre: Skipped 3 previous similar messages [ 7909.267295] Lustre: lustre-MDT0001: Client f00fb04e-1a2a-481b-ab51-f6ede2a0038d (at 192.168.204.44@tcp) reconnecting [ 7910.986906] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7911.004646] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7911.011067] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7911.021994] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to (at 0@lo) [ 7915.195500] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7917.041567] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7922.904663] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 40 [ 7923.120561] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7928.053780] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 40 [ 7928.238986] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7931.341641] Lustre: Permanently deactivating lustre-MDT0001 [ 7931.369073] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active in log lustre-client [ 7931.376504] Lustre: Skipped 2 previous similar messages [ 7937.643247] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7952.866640] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7952.870348] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7952.876805] Lustre: Skipped 2 previous similar messages [ 7952.887718] Lustre: Skipped 3 previous similar messages [ 7954.409660] Lustre: server umount lustre-MDT0000 complete [ 7957.988415] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7958.005229] LustreError: Skipped 7 previous similar messages [ 7958.892519] LustreError: 251486:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776208129 with bad export cookie 6447365236109648356 [ 7958.898097] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7958.903865] LustreError: 251486:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7959.289627] Lustre: server umount lustre-MDT0001 complete [ 7965.319729] Lustre: server umount lustre-OST0000 complete [ 7980.814917] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 19:09:09 (1776208149) [ 7995.015583] Lustre: server umount lustre-OST0001 complete [ 8002.210312] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 8011.900934] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 8021.571190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8028.488344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8035.056352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8041.666418] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8048.864615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8048.917639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8049.126276] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 8049.134746] Lustre: Skipped 2 previous similar messages [ 8049.159856] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 8049.274198] Lustre: lustre-MDT0000: new disk, initializing [ 8049.430455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8049.452327] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8054.013491] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 8063.716326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8063.790817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8063.865724] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 8063.892629] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 8063.895541] Lustre: Skipped 1 previous similar message [ 8063.961985] Lustre: lustre-MDT0001: new disk, initializing [ 8064.065292] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8064.106963] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8064.120116] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8068.987972] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 8078.737187] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8084.780059] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8091.796155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8091.855497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8091.864737] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 8092.128957] Lustre: lustre-OST0000: new disk, initializing [ 8092.133479] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 8092.207503] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8097.207750] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 8101.466966] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8101.487171] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8107.976796] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8110.529720] LustreError: 260020:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8120.589621] LustreError: 260020:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 awake [ 8120.646904] LustreError: 260020:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8120.744621] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8120.813420] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8120.829597] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 8120.994573] Lustre: lustre-OST0001: new disk, initializing [ 8121.000641] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 8126.217227] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 8129.102859] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8129.119178] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8130.744173] LustreError: 260020:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 awake [ 8136.744660] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 8138.609430] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8144.877491] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8144.881474] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8144.893242] Lustre: Skipped 1 previous similar message [ 8144.893961] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8147.255378] Lustre: server umount lustre-OST0001 complete [ 8149.990709] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8150.005559] LustreError: Skipped 1 previous similar message [ 8165.347987] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8165.365660] Lustre: Skipped 1 previous similar message [ 8165.371747] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8165.382672] Lustre: Skipped 1 previous similar message [ 8170.493396] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8170.509199] Lustre: Skipped 1 previous similar message [ 8170.956589] Lustre: server umount lustre-OST0000 complete [ 8175.586598] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8175.602737] Lustre: Skipped 1 previous similar message [ 8175.609624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8175.618437] Lustre: Skipped 1 previous similar message [ 8185.826499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8185.834141] Lustre: Skipped 3 previous similar messages [ 8189.408312] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 8189.721892] Lustre: server umount lustre-MDT0000 complete [ 8190.947997] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8190.963647] LustreError: Skipped 9 previous similar messages [ 8194.386166] LustreError: 260005:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776208365 with bad export cookie 6447365236109650918 [ 8194.392545] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8194.408365] LustreError: 260005:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8215.306937] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 8217.968771] Key type lgssc unregistered [ 8218.164752] LNet: 265217:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8218.174881] LNet: Removed LNI 192.168.204.144@tcp [ 8218.996160] Key type .llcrypt unregistered [ 8219.003927] Key type ._llcrypt unregistered [ 8252.821848] alg: No test for adler32 (adler32-zlib) [ 8253.585727] Key type ._llcrypt registered [ 8253.588614] Key type .llcrypt registered [ 8253.767411] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 8271.115905] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 8272.698398] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8273.471110] Lustre: Lustre: Build Version: 2.15.8_1_g00e7d23 [ 8273.838582] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 8273.842250] LNet: Accept secure, port 988 [ 8275.521094] Key type lgssc registered [ 8276.844513] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8286.640292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8294.575327] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8301.670884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8309.274456] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8316.274852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8316.382882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8317.644943] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 8317.699107] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 8317.847789] Lustre: lustre-MDT0000: new disk, initializing [ 8317.986438] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8318.034683] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8325.161375] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 8337.235705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8337.357600] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8337.494341] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 8337.601270] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 8337.607232] Lustre: Skipped 1 previous similar message [ 8337.764355] Lustre: lustre-MDT0001: new disk, initializing [ 8337.919970] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8337.962850] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8337.966865] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8343.416623] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 8356.242601] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8363.501737] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8370.194710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8370.266656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8370.273708] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 8370.536451] Lustre: lustre-OST0000: new disk, initializing [ 8370.545617] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 8370.635912] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8375.422516] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 8380.955290] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8380.961766] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8384.102408] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8389.328598] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 8389.533545] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8394.082079] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 8394.353253] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8401.381771] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8401.383613] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8401.392061] LustreError: Skipped 1 previous similar message [ 8401.394363] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8401.405455] Lustre: Skipped 1 previous similar message [ 8402.871349] Lustre: server umount lustre-OST0000 complete [ 8409.569566] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8409.583483] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8409.605170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8409.608149] Lustre: Skipped 1 previous similar message [ 8411.619696] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8411.619921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8411.653239] Lustre: Skipped 1 previous similar message [ 8412.877352] Lustre: server umount lustre-MDT0000 complete [ 8416.740634] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8416.765584] LustreError: Skipped 1 previous similar message [ 8417.583605] LustreError: 269350:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776208588 with bad export cookie 13591649562500115734 [ 8417.592547] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8417.604277] LustreError: 269350:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8418.038330] Lustre: server umount lustre-MDT0001 complete [ 8427.202372] Lustre: DEBUG MARKER: == conf-sanity test 52: check recovering objects from lost+found ========================================================== 19:16:36 (1776208596) [ 8435.198595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8435.775306] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8435.886410] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8440.982446] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 8441.321082] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8446.438318] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8449.352340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8449.677200] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8454.909968] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 8463.274837] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8469.491587] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8477.350554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8477.746962] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8482.463810] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 8491.458759] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8515.554552] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8515.565383] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8515.591105] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8517.613413] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8519.179223] Lustre: server umount lustre-OST0000 complete [ 8524.408871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8539.108973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8539.416731] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8544.645518] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 8549.675960] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8549.702869] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.204.144@tcp (at 0@lo) [ 8549.710978] Lustre: lustre-OST0000: deleting orphan objects from 0x0:42 to 0x0:65 [ 8554.217275] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8559.984274] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8560.004044] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.204.144@tcp (at 0@lo) [ 8560.032906] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:65 [ 8570.340620] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8570.340660] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8570.353611] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8570.353620] Lustre: Skipped 2 previous similar messages [ 8570.354557] LustreError: Skipped 1 previous similar message [ 8570.424128] Lustre: Skipped 2 previous similar messages [ 8572.601603] Lustre: server umount lustre-OST0000 complete [ 8580.581821] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8580.586082] Lustre: Skipped 2 previous similar messages [ 8580.598744] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8580.620064] Lustre: Skipped 1 previous similar message [ 8583.524114] Lustre: server umount lustre-MDT0000 complete [ 8585.697218] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8585.724498] LustreError: Skipped 1 previous similar message [ 8587.307845] LustreError: 278548:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776208758 with bad export cookie 13591649562500116588 [ 8587.318658] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8587.327731] LustreError: 278548:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8602.495252] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 8606.028554] Key type lgssc unregistered [ 8606.313397] LNet: 279352:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8606.341364] LNet: Removed LNI 192.168.204.144@tcp [ 8607.144653] Key type .llcrypt unregistered [ 8607.150891] Key type ._llcrypt unregistered [ 8635.566536] alg: No test for adler32 (adler32-zlib) [ 8636.323655] Key type ._llcrypt registered [ 8636.333424] Key type .llcrypt registered [ 8636.465942] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 8638.037713] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8638.566445] Lustre: Lustre: Build Version: 2.15.8_1_g00e7d23 [ 8638.740227] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 8638.743565] LNet: Accept secure, port 988 [ 8640.392239] Key type lgssc registered [ 8641.434760] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8650.334224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8651.831091] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8651.929894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8656.351490] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 8657.378713] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8662.508586] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8665.751795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8666.173031] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8671.295420] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 8679.891783] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8686.698537] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8694.546985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8694.880085] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8699.545860] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 8703.089089] Lustre: lustre-OST0000: deleting orphan objects from 0x0:42 to 0x0:97 [ 8708.970830] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8732.128983] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8732.141768] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8732.160354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8734.177311] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8734.185368] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8734.193695] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8734.213210] Lustre: Skipped 1 previous similar message [ 8737.360799] Lustre: server umount lustre-OST0000 complete [ 8742.898706] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8742.915904] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8742.949673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8748.287143] Lustre: server umount lustre-MDT0000 complete [ 8749.544448] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8749.570358] LustreError: Skipped 1 previous similar message [ 8752.604152] LustreError: 280815:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776208923 with bad export cookie 9808070730748719995 [ 8752.609130] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8752.617294] LustreError: 280815:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8752.889295] Lustre: server umount lustre-MDT0001 complete [ 8768.519168] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 8772.193161] Key type lgssc unregistered [ 8772.479031] LNet: 285269:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8772.500440] LNet: Removed LNI 192.168.204.144@tcp [ 8773.416221] Key type .llcrypt unregistered [ 8773.418052] Key type ._llcrypt unregistered [ 8793.271803] alg: No test for adler32 (adler32-zlib) [ 8794.031479] Key type ._llcrypt registered [ 8794.036531] Key type .llcrypt registered [ 8794.163600] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 8794.880819] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8795.532892] Lustre: Lustre: Build Version: 2.15.8_1_g00e7d23 [ 8795.676747] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 8795.682407] LNet: Accept secure, port 988 [ 8797.288145] Key type lgssc registered [ 8797.945567] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8805.172168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8806.685475] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8806.812626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8811.993767] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 8812.003566] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8817.122184] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8822.253860] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8822.882661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8823.272300] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8827.975463] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 8837.861844] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8845.629996] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8854.410583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8855.152100] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8858.229878] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 8859.978719] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 8868.690941] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8884.196643] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8884.204054] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8884.215499] LustreError: Skipped 1 previous similar message [ 8884.222767] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8884.224474] Lustre: Skipped 1 previous similar message [ 8884.246311] Lustre: Skipped 1 previous similar message [ 8889.316421] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8889.330189] Lustre: Skipped 1 previous similar message [ 8889.711036] Lustre: server umount lustre-OST0000 complete [ 8894.434371] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8894.455793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8899.555558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8899.561156] Lustre: Skipped 2 previous similar messages [ 8900.389701] Lustre: server umount lustre-MDT0000 complete [ 8904.677868] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8904.700610] LustreError: Skipped 1 previous similar message [ 8905.096169] LustreError: 286537:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776209075 with bad export cookie 8285884145095392269 [ 8905.108643] LustreError: 286537:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8905.110316] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8905.655568] Lustre: server umount lustre-MDT0001 complete [ 8920.287602] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 8923.369350] Key type lgssc unregistered [ 8923.612698] LNet: 290389:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8923.626786] LNet: Removed LNI 192.168.204.144@tcp [ 8924.399226] Key type .llcrypt unregistered [ 8924.405331] Key type ._llcrypt unregistered [ 8942.584882] alg: No test for adler32 (adler32-zlib) [ 8943.341620] Key type ._llcrypt registered [ 8943.343899] Key type .llcrypt registered [ 8943.487636] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 8945.303803] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8945.971890] Lustre: Lustre: Build Version: 2.15.8_1_g00e7d23 [ 8946.206915] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 8946.214131] LNet: Accept secure, port 988 [ 8947.867602] Key type lgssc registered [ 8949.254412] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8958.846191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8960.401679] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8960.512063] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8965.607809] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8966.606066] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 8970.749959] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8975.847521] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8977.213668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8977.500906] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8982.345888] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 8992.788181] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9000.146835] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9008.131164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9008.518668] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9010.550124] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 9014.515881] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 9024.165629] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9035.233962] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9035.248391] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9035.284349] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9036.261993] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9036.267555] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9036.268972] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9036.283912] Lustre: Skipped 1 previous similar message [ 9040.110023] Lustre: server umount lustre-OST0000 complete [ 9046.497411] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9046.525153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9046.534930] Lustre: Skipped 1 previous similar message [ 9050.850568] Lustre: server umount lustre-MDT0000 complete [ 9051.618200] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9051.635776] LustreError: Skipped 1 previous similar message [ 9055.646976] LustreError: 291658:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776209226 with bad export cookie 5382010483663552283 [ 9055.662711] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9055.665293] LustreError: 291658:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9056.087289] Lustre: server umount lustre-MDT0001 complete [ 9071.734167] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 9074.421449] Key type lgssc unregistered [ 9074.770177] LNet: 295338:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9074.793593] LNet: Removed LNI 192.168.204.144@tcp [ 9075.615856] Key type .llcrypt unregistered [ 9075.619852] Key type ._llcrypt unregistered [ 9107.639856] alg: No test for adler32 (adler32-zlib) [ 9108.394856] Key type ._llcrypt registered [ 9108.397495] Key type .llcrypt registered [ 9108.566201] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 9109.958987] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9110.274905] Lustre: Lustre: Build Version: 2.15.8_1_g00e7d23 [ 9110.434586] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 9110.440462] LNet: Accept secure, port 988 [ 9112.130557] Key type lgssc registered [ 9113.233634] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9122.245247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9123.835252] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9123.904232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9128.958678] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9129.664891] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 9134.050031] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9138.779529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9139.070479] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9143.809635] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 9152.817293] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9159.623718] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9167.510836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9167.859791] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9173.110083] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 9175.033538] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:193 [ 9182.603465] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9214.944960] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9214.955717] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9214.986733] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9215.972020] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9215.972306] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9218.283528] Lustre: server umount lustre-OST0000 complete [ 9226.210281] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9226.221966] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9226.231582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9226.231590] Lustre: Skipped 1 previous similar message [ 9226.260178] Lustre: Skipped 1 previous similar message [ 9228.440394] Lustre: server umount lustre-MDT0000 complete [ 9231.330145] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9231.348260] LustreError: Skipped 1 previous similar message [ 9232.278654] LustreError: 297610:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776209403 with bad export cookie 10473380796976430913 [ 9232.281464] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9232.299794] LustreError: 297610:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9232.538621] Lustre: server umount lustre-MDT0001 complete [ 9246.105770] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 9249.166835] Key type lgssc unregistered [ 9249.566912] LNet: 301305:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9249.579062] LNet: Removed LNI 192.168.204.144@tcp [ 9250.467857] Key type .llcrypt unregistered [ 9250.470429] Key type ._llcrypt unregistered [ 9269.841853] alg: No test for adler32 (adler32-zlib) [ 9270.604451] Key type ._llcrypt registered [ 9270.616339] Key type .llcrypt registered [ 9270.791782] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 9272.089391] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9272.882358] Lustre: Lustre: Build Version: 2.15.8_1_g00e7d23 [ 9273.128248] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 9273.133560] LNet: Accept secure, port 988 [ 9274.865313] Key type lgssc registered [ 9276.172771] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9284.656186] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9286.140701] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9286.210974] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9291.069854] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 9291.238623] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9296.358662] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9299.755876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9300.226327] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9305.138290] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 9314.054713] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9321.203145] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9327.684525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9328.102860] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9332.697906] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 9335.282539] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:225 [ 9341.015274] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9355.745189] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9355.751621] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9355.755805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9355.768368] Lustre: Skipped 1 previous similar message [ 9355.776920] Lustre: Skipped 1 previous similar message [ 9359.283349] Lustre: server umount lustre-OST0000 complete [ 9365.986456] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9365.991313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9366.010029] Lustre: Skipped 1 previous similar message [ 9366.024611] Lustre: Skipped 1 previous similar message [ 9369.886446] Lustre: server umount lustre-MDT0000 complete [ 9371.116087] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9371.132147] LustreError: Skipped 1 previous similar message [ 9374.137488] LustreError: 302576:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776209544 with bad export cookie 472033520838088344 [ 9374.146411] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9374.161108] LustreError: 302576:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9374.553758] Lustre: server umount lustre-MDT0001 complete [ 9387.171971] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 9390.188926] Key type lgssc unregistered [ 9390.492706] LNet: 306435:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9390.509669] LNet: Removed LNI 192.168.204.144@tcp [ 9391.445246] Key type .llcrypt unregistered [ 9391.447744] Key type ._llcrypt unregistered [ 9409.734095] alg: No test for adler32 (adler32-zlib) [ 9410.490730] Key type ._llcrypt registered [ 9410.499990] Key type .llcrypt registered [ 9410.625891] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 9412.160927] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9412.792959] Lustre: Lustre: Build Version: 2.15.8_1_g00e7d23 [ 9413.037953] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 9413.044421] LNet: Accept secure, port 988 [ 9414.672141] Key type lgssc registered [ 9415.832039] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9424.541703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9426.148889] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9426.213143] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9430.695636] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 9431.530149] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9436.664786] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9440.241306] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9440.863840] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9446.285384] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 9455.267272] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9462.237754] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9469.624894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9469.934634] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9474.320769] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 9477.144770] Lustre: lustre-OST0000: deleting orphan objects from 0x0:227 to 0x0:257 [ 9483.325170] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9491.428092] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9491.442962] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9491.466119] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9492.452520] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9492.460950] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9492.471770] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9492.500072] Lustre: Skipped 1 previous similar message [ 9497.578241] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9497.589553] Lustre: Skipped 1 previous similar message [ 9502.694406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9502.704045] Lustre: Skipped 1 previous similar message [ 9505.248483] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 9505.476777] Lustre: server umount lustre-OST0000 complete [ 9512.419683] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9512.428712] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9512.440966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9523.170345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9523.177038] Lustre: Skipped 4 previous similar messages [ 9524.195514] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 9524.360260] Lustre: server umount lustre-MDT0000 complete [ 9528.292346] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9528.306145] LustreError: Skipped 1 previous similar message [ 9528.580725] LustreError: 307707:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776209699 with bad export cookie 17237896547033087970 [ 9528.585394] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9528.594927] LustreError: 307707:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9528.972401] Lustre: server umount lustre-MDT0001 complete [ 9542.655800] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 9546.186943] Key type lgssc unregistered [ 9546.545726] LNet: 311386:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9546.562880] LNet: Removed LNI 192.168.204.144@tcp [ 9547.685790] Key type .llcrypt unregistered [ 9547.691958] Key type ._llcrypt unregistered [ 9562.287910] alg: No test for adler32 (adler32-zlib) [ 9563.055807] Key type ._llcrypt registered [ 9563.058851] Key type .llcrypt registered [ 9563.183365] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 9581.870722] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 9596.984478] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 9598.067397] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9598.647823] Lustre: Lustre: Build Version: 2.15.8_1_g00e7d23 [ 9598.962335] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 9598.971138] LNet: Accept secure, port 988 [ 9600.656157] Key type lgssc registered [ 9601.976591] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9612.147552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9618.954090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9625.284671] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9631.416729] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9639.083285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9639.189250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9640.444477] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 9640.485214] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 9640.550114] Lustre: lustre-MDT0000: new disk, initializing [ 9640.639428] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9640.652898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9645.896826] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 9657.436449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9657.555659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9657.642891] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 9657.680037] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 9657.687195] Lustre: Skipped 1 previous similar message [ 9657.743741] Lustre: lustre-MDT0001: new disk, initializing [ 9657.825851] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9657.866801] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9657.893432] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9663.398789] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 9673.361813] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9679.731710] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9686.756776] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9686.836961] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9686.852482] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 9687.309883] Lustre: lustre-OST0000: new disk, initializing [ 9687.316846] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 9687.411345] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9693.093221] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 9693.779340] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9693.786851] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9703.581404] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9709.121495] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 9709.329130] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9714.698620] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 9714.938105] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9719.783680] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9719.784074] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9719.807647] LustreError: Skipped 1 previous similar message [ 9719.809847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9719.821975] Lustre: Skipped 1 previous similar message [ 9719.842545] Lustre: Skipped 1 previous similar message [ 9723.355925] Lustre: server umount lustre-OST0000 complete [ 9729.516451] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9729.534632] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9729.557333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9734.001473] Lustre: server umount lustre-MDT0000 complete [ 9735.139722] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9735.160781] LustreError: Skipped 1 previous similar message [ 9736.169373] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9736.203291] LustreError: Skipped 1 previous similar message [ 9738.258781] LustreError: 316012:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776209909 with bad export cookie 15219627867178749443 [ 9738.261425] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9738.273419] LustreError: 316012:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9738.665731] Lustre: server umount lustre-MDT0001 complete [ 9748.337862] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 19:38:37 (1776209917) [ 9755.835929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9756.350651] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9756.430310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9761.252887] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 9761.766425] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9766.881538] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9768.844233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9769.020524] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9774.473204] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 9782.754625] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9789.951683] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9797.250748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9797.694618] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9803.208023] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 9812.184243] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9822.178483] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9822.188541] Lustre: Skipped 1 previous similar message [ 9822.197979] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9822.205339] Lustre: Skipped 3 previous similar messages [ 9827.298591] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9827.311263] Lustre: Skipped 1 previous similar message [ 9827.580506] Lustre: server umount lustre-OST0000 complete [ 9832.418629] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9832.445452] Lustre: Skipped 2 previous similar messages [ 9832.446281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9832.484387] Lustre: Skipped 1 previous similar message [ 9838.240048] Lustre: server umount lustre-MDT0000 complete [ 9842.191697] LustreError: 319696:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776210012 with bad export cookie 15219627867178750297 [ 9842.207653] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9842.225061] LustreError: 319696:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9855.310859] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 9858.248885] Key type lgssc unregistered [ 9858.550054] LNet: 323379:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9858.558366] LNet: Removed LNI 192.168.204.144@tcp [ 9859.616879] Key type .llcrypt unregistered [ 9859.630957] Key type ._llcrypt unregistered [ 9890.544786] alg: No test for adler32 (adler32-zlib) [ 9891.302161] Key type ._llcrypt registered [ 9891.310378] Key type .llcrypt registered [ 9891.423287] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 9903.189875] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 9904.111989] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9904.404737] Lustre: Lustre: Build Version: 2.15.8_1_g00e7d23 [ 9904.565905] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 9904.569500] LNet: Accept secure, port 988 [ 9906.216110] Key type lgssc registered [ 9907.105744] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9915.116856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9922.139643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9927.764337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9933.002261] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9938.104157] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9943.495277] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9950.842604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9950.901243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9952.073783] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 9952.098465] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 9952.164190] Lustre: lustre-MDT0000: new disk, initializing [ 9952.248349] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9952.263698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9957.092906] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 9968.771702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9968.885672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9969.003466] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 9969.099543] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 9969.109730] Lustre: Skipped 1 previous similar message [ 9969.190601] Lustre: lustre-MDT0001: new disk, initializing [ 9969.316802] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9969.359054] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9969.369893] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9973.999249] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [ 9985.346561] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9994.264693] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10002.962415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10003.138069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10003.151648] Lustre: lustre-OST2710-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [10003.516563] Lustre: lustre-OST2710: new disk, initializing [10003.527171] Lustre: srv-lustre-OST2710: No data found on store. Initialize space: rc = -61 [10003.648867] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [10009.166663] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [10012.313029] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [10012.344450] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [10020.452557] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [10027.983198] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10028.062440] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10028.068512] Lustre: lustre-OST03e8-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [10028.150431] Lustre: lustre-OST03e8: new disk, initializing [10028.154451] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space: rc = -61 [10028.251622] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [10033.605321] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [10034.770848] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [10034.782985] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [10042.032902] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [10048.810757] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 40 [10049.004292] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10053.626317] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 40 [10053.822826] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10058.348423] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 40 [10058.704765] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10064.042921] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 40 [10064.270383] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10071.015306] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10071.019793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10071.031063] Lustre: Skipped 3 previous similar messages [10071.049905] Lustre: Skipped 3 previous similar messages [10071.091665] Lustre: server umount lustre-MDT0000 complete [10075.308340] LustreError: 327327:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776210246 with bad export cookie 3192018487317651532 [10075.317456] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10075.673713] Lustre: server umount lustre-MDT0001 complete [10083.235979] Lustre: 325127:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776210246/real 1776210246] req@00000000d5595d4d x1862491255636480/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776210253 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [10083.277531] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10085.344294] Lustre: 332822:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776210250/real 1776210250] req@000000007ffabe63 x1862491255636608/t0(0) o39->lustre-MDT0001-lwp-OST2710@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776210256 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [10085.429965] Lustre: server umount lustre-OST2710 complete [10086.368146] Lustre: 325126:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776210250/real 1776210250] req@00000000cf4da17a x1862491255636672/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776210257 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [10089.978322] Lustre: server umount lustre-OST03e8 complete [10111.526398] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [10122.043710] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [10132.729729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10141.501571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10148.945791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10156.565855] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10165.226869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10165.318211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10165.727950] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [10165.800231] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [10165.941266] Lustre: lustre-MDT0000: new disk, initializing [10166.126624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10166.210947] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10174.147897] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [10190.629378] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10190.731347] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10190.899497] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [10190.986129] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [10190.993429] Lustre: Skipped 1 previous similar message [10191.184532] Lustre: lustre-MDT0001: new disk, initializing [10191.369727] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10191.428496] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10191.442935] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10196.512896] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [10207.617948] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10215.259525] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10222.806107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10222.937765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10222.949631] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [10223.230588] Lustre: lustre-OST0000: new disk, initializing [10223.235400] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [10223.286233] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10229.049676] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [10233.457543] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10233.471049] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10240.719608] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10247.151863] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [10247.505590] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10252.018601] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [10252.249354] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10254.313374] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10254.315263] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10254.315381] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10254.324245] Lustre: Skipped 1 previous similar message [10259.426798] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10259.443876] Lustre: Skipped 2 previous similar messages [10260.333381] Lustre: server umount lustre-OST0000 complete [10268.140232] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10268.155772] LustreError: Skipped 1 previous similar message [10268.166484] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10268.189643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10274.786590] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10274.791294] Lustre: Skipped 3 previous similar messages [10279.393882] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [10279.485622] Lustre: server umount lustre-MDT0000 complete [10279.909810] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10279.929567] LustreError: Skipped 1 previous similar message [10280.929689] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10280.937961] LustreError: Skipped 1 previous similar message [10283.929692] LustreError: 335931:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776210454 with bad export cookie 3192018487317653142 [10283.931168] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10283.959379] LustreError: 335931:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 7 previous similar messages [10296.016954] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 19:47:44 (1776210464) [10298.218688] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [10300.347695] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 19:47:49 (1776210469) [10304.546625] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [10311.412297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10315.004218] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10318.788427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10323.582778] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10325.156948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10333.582312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10334.084602] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10334.116418] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [10334.360846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10339.851668] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [10350.484320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10350.553510] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10350.609266] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [10356.262757] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [10364.695699] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10371.240781] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10378.052464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10378.151645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10378.363963] Lustre: Denying initial registration attempt from nid 192.168.204.144@tcp, specified as failover [10378.372196] LustreError: 160-7: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [10378.380768] LustreError: 343964:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -99 [10378.391534] LustreError: 343964:0:(obd_mount_server.c:1644:server_put_super()) no obd lustre-OST0000 [10378.401832] LustreError: 343964:0:(obd_mount_server.c:131:server_deregister_mount()) lustre-OST0000 not registered [10378.535360] Lustre: server umount lustre-OST0000 complete [10378.544885] Lustre: Skipped 1 previous similar message [10378.554304] LustreError: 343964:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -99 [10384.869796] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10384.883415] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10384.891208] Lustre: Skipped 2 previous similar messages [10384.923143] Lustre: Skipped 1 previous similar message [10389.992646] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10390.013879] LustreError: Skipped 1 previous similar message [10393.072171] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10393.945357] LustreError: 342004:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776210564 with bad export cookie 3192018487317653996 [10393.966379] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10393.970485] LustreError: 342004:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [10408.648931] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [10411.400406] Key type lgssc unregistered [10411.664756] LNet: 345039:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10411.673175] LNet: Removed LNI 192.168.204.144@tcp [10412.570334] Key type .llcrypt unregistered [10412.584117] Key type ._llcrypt unregistered [10427.049049] alg: No test for adler32 (adler32-zlib) [10427.813941] Key type ._llcrypt registered [10427.817792] Key type .llcrypt registered [10427.944262] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [10429.798316] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10430.276738] Lustre: Lustre: Build Version: 2.15.8_1_g00e7d23 [10430.497468] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [10430.503609] LNet: Accept secure, port 988 [10432.249741] Key type lgssc registered [10433.536593] Lustre: Echo OBD driver; http://www.lustre.org/ [10441.396714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10447.075414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10451.559857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10455.730836] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10457.289725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10475.089988] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [10484.915599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10486.186380] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10486.209733] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [10486.362270] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10492.169426] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [10501.695219] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10501.852755] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10501.898531] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [10502.090179] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10507.547461] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [10517.141905] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10522.915797] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10530.940538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10531.020934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10531.325332] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10531.504952] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10537.714854] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [10547.841962] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10553.317249] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10553.317704] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10553.334908] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10553.348857] LustreError: Skipped 1 previous similar message [10553.363650] Lustre: Skipped 1 previous similar message [10556.382443] Lustre: server umount lustre-OST0000 complete [10563.555668] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10563.557205] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10563.579252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10563.600014] Lustre: Skipped 2 previous similar messages [10563.626673] Lustre: Skipped 1 previous similar message [10566.442696] Lustre: server umount lustre-MDT0000 complete [10568.679526] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10568.696639] LustreError: Skipped 1 previous similar message [10569.705715] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10569.727710] LustreError: Skipped 1 previous similar message [10570.126730] LustreError: 350833:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776210740 with bad export cookie 14156583524631953661 [10570.128231] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10570.137990] LustreError: 350833:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [10570.454628] Lustre: server umount lustre-MDT0001 complete [10581.884312] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [10585.116518] Key type lgssc unregistered [10585.552634] LNet: 351638:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10585.572530] LNet: Removed LNI 192.168.204.144@tcp [10586.536743] Key type .llcrypt unregistered [10586.549914] Key type ._llcrypt unregistered [10611.754544] alg: No test for adler32 (adler32-zlib) [10612.511514] Key type ._llcrypt registered [10612.514187] Key type .llcrypt registered [10612.719861] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [10614.203459] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10614.701442] Lustre: Lustre: Build Version: 2.15.8_1_g00e7d23 [10615.020066] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [10615.023800] LNet: Accept secure, port 988 [10616.760159] Key type lgssc registered [10618.368521] Lustre: Echo OBD driver; http://www.lustre.org/ [10627.676692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10629.122213] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10629.183260] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10634.209869] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10634.934872] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [10639.336435] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10643.439523] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10645.245717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10645.994516] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10654.402925] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [10666.544240] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10675.138075] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10682.862908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10683.224988] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10688.398586] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [10697.850752] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10717.670538] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10717.679214] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10717.706607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10719.135046] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.44@tcp (stopping) [10719.204884] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10719.235576] Lustre: Skipped 1 previous similar message [10723.701154] Lustre: server umount lustre-MDT0000 complete [10724.205861] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10728.673896] LustreError: 353082:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776210899 with bad export cookie 5989671926881639546 [10728.686399] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10728.688646] LustreError: 353082:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [10729.371292] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.44@tcp (stopping) [10729.377818] Lustre: Skipped 3 previous similar messages [10729.447465] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10731.503225] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.44@tcp (stopping) [10731.539415] Lustre: Skipped 1 previous similar message [10733.474541] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10733.506347] LustreError: Skipped 6 previous similar messages [10735.163820] Lustre: server umount lustre-MDT0001 complete [10742.567194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10751.324400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10762.230918] Lustre: Evicted from MGS (at 192.168.204.144@tcp) after server handle changed from 0x531f96e202e8dc7a to 0x531f96e202e8f9f4 [10762.259856] Lustre: MGC192.168.204.144@tcp: Connection restored to 192.168.204.144@tcp (at 0@lo) [10762.683799] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10762.693644] LustreError: Skipped 2 previous similar messages [10762.736237] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10762.792296] Lustre: lustre-OST0000: deleting orphan objects from 0x0:166 to 0x0:193 [10764.775675] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [10768.462126] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [10778.884313] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10779.394924] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10783.689961] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [10785.772969] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [10792.506885] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10794.597176] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10801.231773] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10803.562876] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10810.343057] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10810.359164] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10810.389773] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10810.400702] Lustre: Skipped 3 previous similar messages [10815.488171] Lustre: server umount lustre-OST0000 complete [10820.578019] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10820.587883] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10820.598345] Lustre: Skipped 1 previous similar message [10820.602315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10820.605145] Lustre: Skipped 2 previous similar messages [10825.782807] Lustre: server umount lustre-MDT0000 complete [10826.721635] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10826.746732] LustreError: Skipped 5 previous similar messages [10829.710898] LustreError: 353081:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776211000 with bad export cookie 5989671926881647092 [10829.714471] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10829.728559] LustreError: 353081:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [10830.118267] Lustre: server umount lustre-MDT0001 complete [10843.306089] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [10846.335969] Key type lgssc unregistered [10846.797810] LNet: 359630:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10846.811286] LNet: Removed LNI 192.168.204.144@tcp [10847.630853] Key type .llcrypt unregistered [10847.634849] Key type ._llcrypt unregistered [10873.310814] alg: No test for adler32 (adler32-zlib) [10874.065428] Key type ._llcrypt registered [10874.068534] Key type .llcrypt registered [10874.199708] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [10876.474037] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10877.282391] Lustre: Lustre: Build Version: 2.15.8_1_g00e7d23 [10877.597609] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [10877.602588] LNet: Accept secure, port 988 [10879.328283] Key type lgssc registered [10880.634720] Lustre: Echo OBD driver; http://www.lustre.org/ [10890.988070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10892.385710] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10892.452330] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10897.298230] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [10897.890417] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10903.012119] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10907.942975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10908.131846] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10908.547373] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10914.738176] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [10924.638212] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10932.314483] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10939.106400] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10939.126142] Lustre: Skipped 1 previous similar message [10939.128340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10942.594597] Lustre: server umount lustre-MDT0000 complete [10944.482198] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10944.507716] LustreError: Skipped 1 previous similar message [10946.675923] LustreError: 361095:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776211117 with bad export cookie 740117630607341896 [10946.688794] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10947.067851] Lustre: server umount lustre-MDT0001 complete [10963.094541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10963.424645] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10963.501081] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [10963.788394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10969.097131] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [10979.104839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10979.293120] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10979.371089] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [10979.796484] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10985.562638] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [10994.808333] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11003.290796] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11013.647491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11013.832367] LustreError: 13b-9: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [11013.840849] LustreError: 160-7: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [11013.850521] LustreError: 365807:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -2 [11013.855701] LustreError: 365807:0:(obd_mount_server.c:1644:server_put_super()) no obd lustre-OST0000 [11013.860178] LustreError: 365807:0:(obd_mount_server.c:131:server_deregister_mount()) lustre-OST0000 not registered [11013.963706] Lustre: server umount lustre-OST0000 complete [11013.967876] LustreError: 365807:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -2 [11022.410623] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11022.656584] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11022.882330] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11029.468264] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [11030.743072] Lustre: lustre-OST0000: deleting orphan objects from 0x0:166 to 0x0:225 [11040.271146] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11050.856388] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11050.949972] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11050.961579] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [11051.070436] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [11051.146294] Lustre: lustre-OST0001: new disk, initializing [11051.153288] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [11051.274196] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11057.169877] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [11059.382699] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11059.397279] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11067.615276] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [11075.071995] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11075.079496] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11075.080398] Lustre: Skipped 1 previous similar message [11075.082498] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11075.122831] Lustre: Skipped 2 previous similar messages [11077.467462] Lustre: server umount lustre-OST0001 complete [11080.175130] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11080.200914] LustreError: Skipped 1 previous similar message [11082.212043] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11082.220679] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11082.230728] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11085.299746] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11085.314508] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11088.382790] Lustre: server umount lustre-OST0000 complete [11095.521469] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11095.526885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11095.543939] Lustre: Skipped 1 previous similar message [11095.556832] Lustre: Skipped 2 previous similar messages [11099.396306] Lustre: server umount lustre-MDT0000 complete [11100.647478] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11100.674489] LustreError: Skipped 3 previous similar messages [11104.385699] LustreError: 363804:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776211275 with bad export cookie 740117630607342456 [11104.392379] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11104.397771] LustreError: 363804:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [11118.941860] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [11122.407758] Key type lgssc unregistered [11122.811725] LNet: 369194:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11122.837529] LNet: Removed LNI 192.168.204.144@tcp [11123.922616] Key type .llcrypt unregistered [11123.926969] Key type ._llcrypt unregistered [11168.561273] alg: No test for adler32 (adler32-zlib) [11169.320333] Key type ._llcrypt registered [11169.328495] Key type .llcrypt registered [11169.431881] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [11188.320868] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [11189.553690] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11190.449432] Lustre: Lustre: Build Version: 2.15.8_1_g00e7d23 [11190.772066] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [11190.787266] LNet: Accept secure, port 988 [11192.520149] Key type lgssc registered [11193.804718] Lustre: Echo OBD driver; http://www.lustre.org/ [11203.752099] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11211.618648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11218.286664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11225.889684] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11233.067156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11233.141772] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11234.417246] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [11234.450915] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [11234.552676] Lustre: lustre-MDT0000: new disk, initializing [11234.627979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11234.643066] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11238.807123] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [11249.678305] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11249.765504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11249.840407] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [11249.911480] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [11249.919325] Lustre: Skipped 1 previous similar message [11250.030211] Lustre: lustre-MDT0001: new disk, initializing [11250.119221] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11250.161715] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11250.191155] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11254.826448] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [11264.023871] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11269.729823] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11277.520345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11277.593091] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11277.599342] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [11277.873390] Lustre: lustre-OST0000: new disk, initializing [11277.882509] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [11278.006570] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11283.560103] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [11294.604110] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11299.899229] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [11300.122623] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11304.939890] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [11305.324278] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11309.031154] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11309.032983] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11309.050901] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11309.062751] Lustre: Skipped 1 previous similar message [11309.071922] Lustre: Skipped 1 previous similar message [11314.149316] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11314.159069] Lustre: Skipped 1 previous similar message [11319.294645] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11319.305928] Lustre: Skipped 1 previous similar message [11322.338263] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [11322.578990] Lustre: server umount lustre-OST0000 complete [11329.512212] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11329.530150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11329.547422] Lustre: Skipped 1 previous similar message [11334.369516] Lustre: server umount lustre-MDT0000 complete [11334.629637] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11337.730219] LustreError: 373311:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776211508 with bad export cookie 6289616124367082380 [11337.736978] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11337.743048] LustreError: 373311:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [11338.015368] Lustre: server umount lustre-MDT0001 complete [11347.294685] Lustre: DEBUG MARKER: == conf-sanity test 60: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 20:05:16 (1776211516) [11354.000633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11360.169989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11385.836692] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [11397.223601] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [11407.758723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11415.389419] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11422.693570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11430.308874] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11438.521136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11438.610165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11438.902958] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [11438.929978] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [11439.021138] Lustre: lustre-MDT0000: new disk, initializing [11439.123073] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11439.150930] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11444.564808] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [11456.528293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11456.689453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11456.859585] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [11456.941440] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [11456.955199] Lustre: Skipped 1 previous similar message [11457.099268] Lustre: lustre-MDT0001: new disk, initializing [11457.246218] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11457.374091] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11457.389464] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11465.342858] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [11479.668562] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11487.873354] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11496.147211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11496.280820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11496.288655] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [11496.553377] Lustre: lustre-OST0000: new disk, initializing [11496.560885] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [11496.632284] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11502.226569] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11502.247771] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11502.314857] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [11516.056318] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11521.283898] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [11521.536978] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11527.036716] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [11527.331321] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11533.292413] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11533.297508] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11533.321644] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11533.324665] LustreError: Skipped 1 previous similar message [11533.340509] Lustre: Skipped 2 previous similar messages [11543.538404] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11543.548136] Lustre: Skipped 3 previous similar messages [11544.032527] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [11544.438903] Lustre: server umount lustre-OST0000 complete [11549.670411] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11549.680667] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11549.695835] Lustre: Skipped 1 previous similar message [11553.761810] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11555.666678] Lustre: server umount lustre-MDT0000 complete [11558.881059] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11558.888667] LustreError: Skipped 2 previous similar messages [11560.175204] LustreError: 383174:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776211730 with bad export cookie 6289616124367083234 [11560.181878] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11560.193192] LustreError: 383174:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [11570.781944] Lustre: DEBUG MARKER: == conf-sanity test 61: large xattr ====================== 20:08:59 (1776211739) [11580.285328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11580.684294] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11580.736387] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11586.021898] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11586.187414] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [11591.159803] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11595.155749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11600.756976] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [11610.899516] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11617.960577] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11628.039641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11628.309943] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11628.317782] Lustre: Skipped 1 previous similar message [11635.362176] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [11645.500934] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11653.907121] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61.conf-sanity [11656.301750] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61.conf-sanity [11658.771712] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61.conf-sanity [11660.910299] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61.conf-sanity after remounting MDS [11663.410654] Lustre: Failing over lustre-MDT0000 [11663.553908] Lustre: server umount lustre-MDT0000 complete [11663.556094] Lustre: Skipped 1 previous similar message [11666.405319] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11666.408046] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11666.435392] Lustre: Skipped 2 previous similar messages [11666.468569] LustreError: Skipped 2 previous similar messages [11673.568151] Lustre: 371538:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776211837/real 1776211837] req@00000000318dda7e x1862492604131968/t0(0) o400->MGC192.168.204.144@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776211844 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [11673.606202] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11684.231901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11684.833636] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11684.833636] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11684.833645] LustreError: Skipped 17 previous similar messages [11684.882192] LustreError: Skipped 1 previous similar message [11690.984036] Lustre: Evicted from MGS (at 192.168.204.144@tcp) after server handle changed from 0x5749348a92781a38 to 0x5749348a92781e21 [11690.996491] Lustre: MGC192.168.204.144@tcp: Connection restored to 192.168.204.144@tcp (at 0@lo) [11691.302743] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11691.374168] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11696.442560] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [11696.643593] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [11696.698622] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [11696.777692] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [11707.586197] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11709.408662] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11713.375667] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61.conf-sanity [11715.901186] Lustre: Failing over lustre-MDT0000 [11716.081672] Lustre: server umount lustre-MDT0000 complete [11717.089942] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11717.102183] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11717.116697] Lustre: Skipped 2 previous similar messages [11724.257557] Lustre: 371537:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776211887/real 1776211887] req@0000000020a8f2dd x1862492604143104/t0(0) o400->MGC192.168.204.144@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776211894 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [11724.293565] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11737.356856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11741.668874] Lustre: Evicted from MGS (at 192.168.204.144@tcp) after server handle changed from 0x5749348a92781e21 to 0x5749348a92782131 [11741.687243] Lustre: MGC192.168.204.144@tcp: Connection restored to (at 0@lo) [11741.697486] Lustre: Skipped 2 previous similar messages [11742.065698] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11742.076486] Lustre: Skipped 1 previous similar message [11742.134476] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11745.651220] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11746.934601] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [11747.303844] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [11747.326621] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [11747.366473] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:97 [11757.804613] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11759.773374] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11767.786508] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11767.788545] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11767.789269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11767.789275] Lustre: Skipped 3 previous similar messages [11767.796104] Lustre: Skipped 1 previous similar message [11772.295715] Lustre: server umount lustre-MDT0000 complete [11772.901952] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11772.920523] LustreError: Skipped 40 previous similar messages [11776.439371] LustreError: 385727:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776211947 with bad export cookie 6289616124367085873 [11776.443667] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11776.449690] LustreError: 385727:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [11786.720246] Lustre: 391544:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776211951/real 1776211951] req@00000000d160dd16 x1862492604156032/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776211957 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [11803.426276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11809.365576] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [11818.281564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11823.434744] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [11832.259896] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11838.483296] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11846.132242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11851.906733] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [11854.766556] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:129 [11860.148279] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11867.113150] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [11867.119150] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11867.129518] Lustre: Skipped 1 previous similar message [11867.134829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11867.139409] Lustre: Skipped 2 previous similar messages [11872.485833] Lustre: server umount lustre-OST0000 complete [11872.494125] Lustre: Skipped 2 previous similar messages [11888.330830] LustreError: 392352:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776212059 with bad export cookie 6289616124367086601 [11888.335539] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11888.341188] LustreError: 392352:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [11901.806456] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [11904.671836] Key type lgssc unregistered [11904.980240] LNet: 396026:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11904.986216] LNet: Removed LNI 192.168.204.144@tcp [11905.893281] Key type .llcrypt unregistered [11905.896213] Key type ._llcrypt unregistered [11933.796713] alg: No test for adler32 (adler32-zlib) [11934.559899] Key type ._llcrypt registered [11934.564447] Key type .llcrypt registered [11934.698426] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [11935.872471] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11936.313249] Lustre: Lustre: Build Version: 2.15.8_1_g00e7d23 [11936.736504] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [11936.741224] LNet: Accept secure, port 988 [11938.400191] Key type lgssc registered [11939.797460] Lustre: Echo OBD driver; http://www.lustre.org/ [11951.584376] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11951.605589] LustreError: 397535:0:(osd_handler.c:8215:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11951.624506] LustreError: 397535:0:(obd_config.c:774:class_setup()) setup lustre-MDT0000-osd failed (-22) [11951.638395] LustreError: 397535:0:(obd_mount.c:200:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11951.653453] LustreError: 397535:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11951.664636] LustreError: 397535:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -22 [11961.123423] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11961.145384] LustreError: 397933:0:(osd_handler.c:8215:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11961.165255] LustreError: 397933:0:(obd_config.c:774:class_setup()) setup lustre-OST0000-osd failed (-22) [11961.180496] LustreError: 397933:0:(obd_mount.c:200:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11961.202379] LustreError: 397933:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11961.215227] LustreError: 397933:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -22 [11984.283602] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [11988.168238] Key type lgssc unregistered [11988.517221] LNet: 398891:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11988.534687] LNet: Removed LNI 192.168.204.144@tcp [11989.735175] Key type .llcrypt unregistered [11989.741519] Key type ._llcrypt unregistered [12018.346114] alg: No test for adler32 (adler32-zlib) [12019.115617] Key type ._llcrypt registered [12019.124491] Key type .llcrypt registered [12019.259242] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [12036.558298] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [12038.294296] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12038.817279] Lustre: Lustre: Build Version: 2.15.8_1_g00e7d23 [12039.190132] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [12039.195911] LNet: Accept secure, port 988 [12040.928182] Key type lgssc registered [12042.515292] Lustre: Echo OBD driver; http://www.lustre.org/ [12053.587445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12063.075937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12071.513263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12080.531837] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12089.006149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12089.114368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12090.395859] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [12090.424665] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [12090.529676] Lustre: lustre-MDT0000: new disk, initializing [12090.667598] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12090.693177] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12096.082946] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [12108.725740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12108.835100] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12109.004611] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [12109.116329] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [12109.125831] Lustre: Skipped 1 previous similar message [12109.253419] Lustre: lustre-MDT0001: new disk, initializing [12109.445985] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12109.509042] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12109.523938] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12115.355901] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [12128.556445] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12136.563732] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12145.198829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12145.271381] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12145.276657] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [12145.520048] Lustre: lustre-OST0000: new disk, initializing [12145.530532] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [12145.633161] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12151.498619] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [12154.428440] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12154.444304] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12162.848536] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12168.244583] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [12168.530649] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12173.879228] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [12174.160803] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12180.450466] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12180.452311] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12180.462491] LustreError: Skipped 1 previous similar message [12180.463135] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12180.494268] Lustre: Skipped 1 previous similar message [12182.712732] Lustre: server umount lustre-OST0000 complete [12190.689253] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12190.714602] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12190.721081] Lustre: Skipped 2 previous similar messages [12193.598255] Lustre: server umount lustre-MDT0000 complete [12195.809571] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12195.829564] LustreError: Skipped 1 previous similar message [12196.859539] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12196.884885] LustreError: Skipped 1 previous similar message [12198.490159] LustreError: 402206:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776212369 with bad export cookie 17984720118328020361 [12198.496926] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12198.502206] LustreError: 402206:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [12198.952487] Lustre: server umount lustre-MDT0001 complete [12210.944438] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [12213.373601] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 20:19:41 (1776212381) [12222.160559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12222.786541] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12222.930042] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12228.070696] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12229.461705] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [12233.201271] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12241.246822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12241.771523] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12248.132697] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [12258.460916] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12265.388505] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12273.508140] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12273.921616] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12279.431237] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [12288.910211] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12297.040256] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12297.131620] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12297.144597] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [12297.352188] Lustre: lustre-OST0001: new disk, initializing [12297.365757] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [12297.471284] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12303.128440] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [12303.605206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12303.616399] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12315.171118] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [12322.205941] Lustre: lustre-OST0001: Not available for connect from 192.168.204.44@tcp (stopping) [12324.322870] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12324.334403] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12324.353679] LustreError: Skipped 1 previous similar message [12324.354485] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [12324.384294] Lustre: Skipped 2 previous similar messages [12326.158148] Lustre: server umount lustre-OST0001 complete [12327.286327] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 192.168.204.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12327.303345] LustreError: Skipped 1 previous similar message [12344.808734] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12344.829020] LustreError: Skipped 7 previous similar messages [12348.918259] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12348.918715] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12348.944212] LustreError: Skipped 1 previous similar message [12348.945036] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12348.945041] Lustre: Skipped 1 previous similar message [12348.989424] Lustre: Skipped 1 previous similar message [12351.808354] Lustre: server umount lustre-OST0000 complete [12359.650869] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12359.659501] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12359.691199] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12359.702526] Lustre: Skipped 3 previous similar messages [12362.710081] Lustre: server umount lustre-MDT0000 complete [12368.088193] LustreError: 406798:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776212538 with bad export cookie 17984720118328021208 [12368.092190] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12368.097679] LustreError: 406798:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [12388.797857] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [12392.116631] Key type lgssc unregistered [12392.474761] LNet: 411728:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12392.483731] LNet: Removed LNI 192.168.204.144@tcp [12393.446917] Key type .llcrypt unregistered [12393.451380] Key type ._llcrypt unregistered [12431.617406] alg: No test for adler32 (adler32-zlib) [12432.370490] Key type ._llcrypt registered [12432.372800] Key type .llcrypt registered [12432.487730] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [12448.469778] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [12449.292781] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12449.745541] Lustre: Lustre: Build Version: 2.15.8_1_g00e7d23 [12450.085517] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [12450.099353] LNet: Accept secure, port 988 [12451.832127] Key type lgssc registered [12452.993423] Lustre: Echo OBD driver; http://www.lustre.org/ [12463.786384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12472.555151] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12483.836613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12495.777093] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12504.291898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12504.394388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12505.794585] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [12505.849595] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [12505.983058] Lustre: lustre-MDT0000: new disk, initializing [12506.110263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12506.135830] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12511.137444] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [12523.107653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12523.196359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12523.364075] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [12523.433924] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [12523.439054] Lustre: Skipped 1 previous similar message [12523.531492] Lustre: lustre-MDT0001: new disk, initializing [12523.641624] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12523.698295] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12523.703575] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12528.843156] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [12540.516466] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12548.380751] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12556.196166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12556.296353] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12556.302269] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [12556.617590] Lustre: lustre-OST0000: new disk, initializing [12556.627857] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [12556.749418] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12561.919924] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [12566.670939] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12566.688488] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12573.087399] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12578.779292] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [12578.947894] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12583.332886] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [12583.502648] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12587.489697] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12587.503272] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12587.536118] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12587.544332] Lustre: Skipped 1 previous similar message [12591.780977] Lustre: server umount lustre-OST0000 complete [12597.732455] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12597.761096] Lustre: Skipped 1 previous similar message [12597.779171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12602.794456] Lustre: server umount lustre-MDT0000 complete [12602.849332] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12602.870237] LustreError: Skipped 1 previous similar message [12603.881705] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12603.902815] LustreError: Skipped 1 previous similar message [12608.101271] LustreError: 416712:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776212778 with bad export cookie 3246417379119255502 [12608.101887] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12608.117777] LustreError: 416712:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [12608.599822] Lustre: server umount lustre-MDT0001 complete [12618.780134] Lustre: DEBUG MARKER: == conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================================================== 20:26:27 (1776212787) [12629.933444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [12639.365456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12639.860338] Lustre: lustre-MDT0000: new disk, initializing [12639.885100] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12639.965566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12645.347465] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12645.718425] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [12650.481779] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12655.746727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12656.193758] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12661.483386] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [12670.544698] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12677.936687] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12679.764656] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12679.777258] Lustre: Skipped 1 previous similar message [12679.793156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12679.805965] Lustre: Skipped 2 previous similar messages [12684.770643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12684.781848] Lustre: Skipped 1 previous similar message [12686.115344] Lustre: server umount lustre-MDT0000 complete [12690.916515] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12690.938395] LustreError: Skipped 2 previous similar messages [12691.059664] LustreError: 420982:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776212861 with bad export cookie 3246417379119256356 [12691.065314] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12691.069578] LustreError: 420982:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [12691.389446] Lustre: server umount lustre-MDT0001 complete [12704.749640] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 20:27:53 (1776212873) [12714.799901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12715.441746] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12715.589695] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12720.042647] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [12729.860077] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12735.768411] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [12745.191726] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12751.882056] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12758.528226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12758.941471] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12758.954122] Lustre: Skipped 1 previous similar message [12765.145793] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [12775.810724] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12784.921110] Lustre: Permanently deactivating lustre-OST0000 [12784.943913] Lustre: Setting parameter lustre-OST0000-osc.osc.active in log lustre-client [12792.924970] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12795.403213] LustreError: 426654:0:(mgs_llog.c:1551:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002020 [12795.407754] LustreError: 426654:0:(mgs_llog.c:1605:mgs_replace_nids()) Only MGS is allowed to be started [12795.413150] LustreError: 426654:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -115 [12798.867156] LustreError: 426703:0:(mgs_llog.c:1605:mgs_replace_nids()) Only MGS is allowed to be started [12798.886804] LustreError: 426703:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -115 [12807.160793] Lustre: server umount lustre-OST0000 complete [12809.766121] LustreError: 426952:0:(mgs_llog.c:1605:mgs_replace_nids()) Only MGS is allowed to be started [12809.776784] LustreError: 426952:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -115 [12812.257656] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12812.270064] LustreError: Skipped 1 previous similar message [12812.274867] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12812.294302] Lustre: Skipped 1 previous similar message [12812.305826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12812.316668] Lustre: Skipped 2 previous similar messages [12817.889458] Lustre: 427049:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776212982/real 1776212982] req@00000000b533b4b7 x1862493924278400/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1776212988 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [12818.402060] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12818.411520] LustreError: Skipped 3 previous similar messages [12822.002370] LustreError: 423586:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776212992 with bad export cookie 3246417379119256909 [12822.008497] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12822.017927] LustreError: 423586:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 7 previous similar messages [12828.128259] Lustre: 427251:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776212992/real 1776212992] req@000000002707953f x1862493924279552/t0(0) o9->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1776212998 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [12828.471376] Lustre: server umount lustre-MDT0001 complete [12828.476149] Lustre: Skipped 1 previous similar message [12836.782126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12840.951610] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [12843.491124] LustreError: 428160:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -6 [12850.439064] Lustre: 428507:0:(mgs_llog.c:1311:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.144@tcp [12858.848423] Lustre: 428606:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776213023/real 1776213023] req@00000000ddfe31d6 x1862493924280064/t0(0) o251->MGC192.168.204.144@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776213029 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [12872.446393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12872.940898] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12872.945935] Lustre: Skipped 1 previous similar message [12872.988283] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12878.498919] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [12883.426630] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12883.450877] LustreError: Skipped 2 previous similar messages [12887.643740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12887.968362] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12893.136785] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [12901.974935] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12908.922311] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12911.415174] Lustre: Permanently reactivating lustre-OST0000 [12911.444450] Lustre: Modifying parameter lustre-OST0000-osc.osc.active in log lustre-client [12911.452876] Lustre: Skipped 2 previous similar messages [12937.363599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12937.829333] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12937.836996] Lustre: Skipped 1 previous similar message [12943.147389] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [12953.618655] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12959.717588] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12959.733303] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12959.735095] Lustre: Skipped 2 previous similar messages [12959.765702] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12959.767886] Lustre: Skipped 3 previous similar messages [12961.623842] Lustre: server umount lustre-OST0000 complete [12961.627029] Lustre: Skipped 1 previous similar message [12969.954203] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12969.957659] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12969.996615] Lustre: Skipped 1 previous similar message [12976.870352] LustreError: 429214:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776213147 with bad export cookie 3246417379119257868 [12976.872296] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12976.880456] LustreError: 429214:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [12990.118066] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [12992.983574] Key type lgssc unregistered [12993.348362] LNet: 433432:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12993.379103] LNet: Removed LNI 192.168.204.144@tcp [12994.344746] Key type .llcrypt unregistered [12994.348430] Key type ._llcrypt unregistered [13014.423716] alg: No test for adler32 (adler32-zlib) [13015.189913] Key type ._llcrypt registered [13015.197958] Key type .llcrypt registered [13015.470676] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [13036.490754] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [13037.585733] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13038.134298] Lustre: Lustre: Build Version: 2.15.8_1_g00e7d23 [13038.543752] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [13038.549588] LNet: Accept secure, port 988 [13040.192186] Key type lgssc registered [13042.195344] Lustre: Echo OBD driver; http://www.lustre.org/ [13053.574100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13061.747919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13069.180717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13075.654051] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13083.926141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13084.051187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13085.371294] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [13085.403238] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [13085.490348] Lustre: lustre-MDT0000: new disk, initializing [13085.589185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13085.608701] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13091.666335] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [13104.165208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13104.267389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13104.412373] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [13104.467825] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [13104.473383] Lustre: Skipped 1 previous similar message [13104.620631] Lustre: lustre-MDT0001: new disk, initializing [13104.734311] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13104.798066] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13104.817452] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13109.826180] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [13120.939372] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13127.268662] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13134.064326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13134.153278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13134.166759] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [13134.440463] Lustre: lustre-OST0000: new disk, initializing [13134.444835] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [13134.609293] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13139.722982] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [13144.186863] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13144.210983] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13152.298690] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13158.470261] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [13158.825179] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13164.296433] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [13164.519993] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13170.149582] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13170.151082] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13170.162607] LustreError: Skipped 1 previous similar message [13170.165379] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13170.175474] Lustre: Skipped 1 previous similar message [13170.191310] Lustre: Skipped 1 previous similar message [13172.621763] Lustre: server umount lustre-OST0000 complete [13180.385689] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13180.389448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13180.406706] Lustre: Skipped 1 previous similar message [13182.856260] Lustre: server umount lustre-MDT0000 complete [13185.505106] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13185.527673] LustreError: Skipped 1 previous similar message [13186.534476] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13186.545082] LustreError: Skipped 1 previous similar message [13187.225489] LustreError: 436729:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776213357 with bad export cookie 2082975241882773680 [13187.238729] LustreError: 436729:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [13187.244951] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13187.691930] Lustre: server umount lustre-MDT0001 complete [13199.830560] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [13202.903031] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 20:36:11 (1776213371) [13212.298096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13213.012742] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13213.109994] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13218.223318] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [13218.281424] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13223.397248] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13228.556555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13229.015400] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13234.092698] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [13242.657125] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13249.846651] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13257.706737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13258.115694] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13264.538773] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [13273.904827] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13276.393169] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [13279.182413] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [13284.833015] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13284.842033] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13284.850836] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13284.853262] Lustre: Skipped 1 previous similar message [13288.930271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13288.941354] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13288.949591] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13290.285427] Lustre: server umount lustre-OST0000 complete [13295.585092] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13295.596978] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13295.623530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13295.625542] Lustre: Skipped 1 previous similar message [13300.814463] Lustre: server umount lustre-MDT0000 complete [13304.296766] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13304.311887] LustreError: Skipped 2 previous similar messages [13304.977582] LustreError: 441318:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776213475 with bad export cookie 2082975241882774534 [13304.994303] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13305.000592] LustreError: 441318:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [13318.783458] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [13322.452433] Key type lgssc unregistered [13322.761322] LNet: 445191:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13322.771839] LNet: Removed LNI 192.168.204.144@tcp [13323.755963] Key type .llcrypt unregistered [13323.759419] Key type ._llcrypt unregistered [13351.983839] alg: No test for adler32 (adler32-zlib) [13352.737651] Key type ._llcrypt registered [13352.741463] Key type .llcrypt registered [13352.881926] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [13353.194769] Key type .llcrypt unregistered [13353.197364] Key type ._llcrypt unregistered [13371.036185] alg: No test for adler32 (adler32-zlib) [13371.801862] Key type ._llcrypt registered [13371.806532] Key type .llcrypt registered [13371.948888] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [13373.124424] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13374.186777] Lustre: Lustre: Build Version: 2.15.8_1_g00e7d23 [13374.669017] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [13374.672716] LNet: Accept secure, port 988 [13376.440222] Key type lgssc registered [13377.755919] Lustre: Echo OBD driver; http://www.lustre.org/ [13387.929259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13390.175662] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13390.278930] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13394.275209] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13398.092270] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [13399.522149] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13404.646531] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13409.366173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13409.765844] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13409.893346] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13414.910894] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [13419.495963] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13419.506571] LustreError: Skipped 4 previous similar messages [13419.515038] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [13422.944922] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13430.245237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13430.710956] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13435.737530] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [13445.089483] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13445.110249] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13445.134974] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13446.164452] Lustre: server umount lustre-OST0000 complete [13450.725236] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [13450.741287] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13450.757515] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13452.252641] Lustre: server umount lustre-MDT0000 complete [13452.259050] Lustre: 446856:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776213616/real 1776213616] req@0000000043212280 x1862494894175104/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1776213623 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [13452.281553] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13452.305176] Lustre: Skipped 1 previous similar message [13452.316692] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13452.339361] LustreError: Skipped 5 previous similar messages [13456.430207] LustreError: 447525:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776213627 with bad export cookie 8325178434222841677 [13456.451420] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13456.451479] LustreError: 447525:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [13456.748793] Lustre: server umount lustre-MDT0001 complete [13470.919914] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [13474.342430] Key type lgssc unregistered [13474.721716] LNet: 450808:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13474.736066] LNet: Removed LNI 192.168.204.144@tcp [13475.840836] Key type .llcrypt unregistered [13475.842355] Key type ._llcrypt unregistered [13504.851873] alg: No test for adler32 (adler32-zlib) [13505.620853] Key type ._llcrypt registered [13505.632207] Key type .llcrypt registered [13505.777962] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [13507.417413] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13507.984580] Lustre: Lustre: Build Version: 2.15.8_1_g00e7d23 [13508.433861] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [13508.443530] LNet: Accept secure, port 988 [13510.184619] Key type lgssc registered [13511.250788] Lustre: Echo OBD driver; http://www.lustre.org/ [13519.442471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13536.224915] LustreError: 452266:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.204.144@tcp: failed processing log, type 1: rc = -5 [13567.968228] LustreError: 452266:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.204.144@tcp: failed processing log, type 4: rc = -110 [13597.728207] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13603.176444] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [13613.193123] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13621.366225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13646.252518] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13646.358654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13646.410266] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:97 [13651.001321] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [13651.440627] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13656.554079] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13661.694405] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13661.703585] LustreError: Skipped 1 previous similar message [13663.332630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13663.554133] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13668.438862] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [13680.325655] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13688.347426] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13699.557085] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13699.564884] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13699.583243] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13699.598900] Lustre: Skipped 1 previous similar message [13704.687088] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13704.692507] Lustre: Skipped 1 previous similar message [13709.796623] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13709.801551] Lustre: Skipped 1 previous similar message [13712.352144] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [13712.556951] Lustre: server umount lustre-OST0000 complete [13720.036207] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13720.036817] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13720.056516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13720.069974] Lustre: Skipped 2 previous similar messages [13720.099279] Lustre: Skipped 1 previous similar message [13725.157119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13725.163914] Lustre: Skipped 1 previous similar message [13731.297988] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [13731.503588] Lustre: server umount lustre-MDT0000 complete [13735.405550] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13735.426895] LustreError: Skipped 1 previous similar message [13736.194302] LustreError: 452272:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776213906 with bad export cookie 11427658290301625450 [13736.199214] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13736.203212] LustreError: 452272:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [13736.596997] Lustre: server umount lustre-MDT0001 complete [13754.198848] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [13758.288269] Key type lgssc unregistered [13758.850123] LNet: 455989:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13758.860309] LNet: Removed LNI 192.168.204.144@tcp [13759.866385] Key type .llcrypt unregistered [13759.868620] Key type ._llcrypt unregistered [13787.638292] alg: No test for adler32 (adler32-zlib) [13788.397161] Key type ._llcrypt registered [13788.400894] Key type .llcrypt registered [13788.567058] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [13789.834371] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13790.381989] Lustre: Lustre: Build Version: 2.15.8_1_g00e7d23 [13790.650398] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [13790.654696] LNet: Accept secure, port 988 [13792.360242] Key type lgssc registered [13794.289271] Lustre: Echo OBD driver; http://www.lustre.org/ [13804.986174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13806.380500] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13806.425977] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13811.680667] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13812.232659] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [13816.813132] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13821.392655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13821.784448] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13826.696937] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [13835.473512] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13845.919851] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13855.087881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13855.720272] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13857.798191] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:129 [13861.106156] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [13870.098362] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13877.190051] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.44@tcp (stopping) [13877.219783] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13877.240676] Lustre: Skipped 1 previous similar message [13880.402831] Lustre: server umount lustre-MDT0000 complete [13882.228735] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13884.059862] LustreError: 457457:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.44@tcp arrived at 1776214054 with bad export cookie 15277000067161976310 [13885.855626] LustreError: 457457:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776214056 with bad export cookie 15277000067161975421 [13885.869779] LustreError: 457457:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [13885.881560] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13887.458355] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13887.459218] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13887.463839] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13887.488424] LustreError: Skipped 4 previous similar messages [13887.553134] Lustre: Skipped 1 previous similar message [13887.581671] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13887.593267] Lustre: Skipped 3 previous similar messages [13890.468527] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13892.088826] Lustre: server umount lustre-OST0000 complete [13896.674573] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13896.693536] LustreError: Skipped 5 previous similar messages [13901.360294] Lustre: server umount lustre-MDT0001 complete [13920.346811] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [13923.277882] Key type lgssc unregistered [13923.538551] LNet: 461305:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13923.554225] LNet: Removed LNI 192.168.204.144@tcp [13924.273913] Key type .llcrypt unregistered [13924.277792] Key type ._llcrypt unregistered [13950.351571] alg: No test for adler32 (adler32-zlib) [13951.105487] Key type ._llcrypt registered [13951.108019] Key type .llcrypt registered [13951.252312] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [13952.257041] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13952.869456] Lustre: Lustre: Build Version: 2.15.8_1_g00e7d23 [13953.149755] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [13953.152869] LNet: Accept secure, port 988 [13954.820240] Key type lgssc registered [13955.926650] Lustre: Echo OBD driver; http://www.lustre.org/ [13963.910898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13965.453614] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13965.540245] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13970.035348] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [13970.914214] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13976.036711] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13978.564073] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13979.026622] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13983.963875] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [13992.796530] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13999.658881] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14007.610932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14008.210729] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14013.258718] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [14015.352758] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:161 [14021.288696] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14025.700084] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14025.719431] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [14025.728385] Lustre: Skipped 1 previous similar message [14028.191376] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.44@tcp (stopping) [14030.820494] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [14031.384446] Lustre: server umount lustre-MDT0001 complete [14033.262134] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 192.168.204.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [14041.057802] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14041.067260] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14041.080633] Lustre: Skipped 1 previous similar message [14041.084423] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14041.103111] LustreError: Skipped 2 previous similar messages [14041.107749] Lustre: Skipped 1 previous similar message [14045.443665] Lustre: server umount lustre-OST0000 complete [14055.912124] Lustre: 465840:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776214220/real 1776214220] req@000000001484611b x1862495500255872/t0(0) o251->MGC192.168.204.144@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776214226 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [14056.230916] Lustre: server umount lustre-MDT0000 complete [14076.281384] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [14079.170941] Key type lgssc unregistered [14079.421186] LNet: 466597:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14079.428174] LNet: Removed LNI 192.168.204.144@tcp [14080.269825] Key type .llcrypt unregistered [14080.273543] Key type ._llcrypt unregistered [14111.425995] alg: No test for adler32 (adler32-zlib) [14112.180276] Key type ._llcrypt registered [14112.187260] Key type .llcrypt registered [14112.344811] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [14131.495280] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [14132.760538] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14133.572212] Lustre: Lustre: Build Version: 2.15.8_1_g00e7d23 [14133.806636] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [14133.813873] LNet: Accept secure, port 988 [14135.488343] Key type lgssc registered [14137.015468] Lustre: Echo OBD driver; http://www.lustre.org/ [14146.988754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14156.156962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14163.850991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14171.582194] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14184.131976] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [14196.436195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14203.820528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14203.911789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14205.153182] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [14205.208528] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [14205.320762] Lustre: lustre-MDT0000: new disk, initializing [14205.441401] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14205.470908] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14211.284174] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [14222.510713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14230.453119] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14230.519732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14230.525302] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [14230.796725] Lustre: lustre-OST0000: new disk, initializing [14230.803908] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [14230.820878] Lustre: Skipped 1 previous similar message [14230.927366] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14237.005398] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [14249.615730] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14259.013847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14265.995527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14266.104961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14266.276544] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [14266.338308] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [14266.516139] Lustre: lustre-MDT0001: new disk, initializing [14266.587209] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [14266.607286] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [14266.666293] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14271.689729] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [14282.455628] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 40 [14282.738932] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [14288.145481] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 40 [14288.385226] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [14293.988854] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14293.991476] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [14293.993409] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [14294.002343] Lustre: Skipped 1 previous similar message [14296.982473] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.44@tcp (stopping) [14296.987305] Lustre: Skipped 1 previous similar message [14298.747996] Lustre: server umount lustre-MDT0001 complete [14299.124345] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14299.154379] LustreError: Skipped 1 previous similar message [14302.078939] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 192.168.204.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [14304.236517] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14304.264483] LustreError: Skipped 1 previous similar message [14309.351233] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14309.359936] LustreError: Skipped 1 previous similar message [14314.465851] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14314.496262] LustreError: Skipped 1 previous similar message [14319.586808] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14319.617972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14324.707423] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14324.707439] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14324.717224] LustreError: Skipped 1 previous similar message [14324.894186] Lustre: server umount lustre-OST0000 complete [14342.624337] Lustre: 475036:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776214507/real 1776214507] req@00000000b85db238 x1862495690052096/t0(0) o251->MGC192.168.204.144@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776214513 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [14342.938367] Lustre: server umount lustre-MDT0000 complete [14360.975308] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [14364.036471] Key type lgssc unregistered [14364.315847] LNet: 475793:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14364.331825] LNet: Removed LNI 192.168.204.144@tcp [14365.198785] Key type .llcrypt unregistered [14365.201861] Key type ._llcrypt unregistered [14411.177452] alg: No test for adler32 (adler32-zlib) [14411.932656] Key type ._llcrypt registered [14411.935899] Key type .llcrypt registered [14412.054329] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [14412.303619] Key type .llcrypt unregistered [14412.305652] Key type ._llcrypt unregistered [14433.165532] alg: No test for adler32 (adler32-zlib) [14433.926739] Key type ._llcrypt registered [14433.933088] Key type .llcrypt registered [14434.098846] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [14435.193850] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14435.776386] Lustre: Lustre: Build Version: 2.15.8_1_g00e7d23 [14436.039410] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [14436.044599] LNet: Accept secure, port 988 [14437.808120] Key type lgssc registered [14439.070609] Lustre: Echo OBD driver; http://www.lustre.org/ [14448.987144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14456.927562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14465.595227] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14475.477919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14475.566528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14477.165360] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [14477.204488] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [14477.433200] Lustre: lustre-MDT0000: new disk, initializing [14477.736223] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14477.759835] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14483.737145] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [14496.016314] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14496.141091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14496.310891] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [14496.415432] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [14496.420816] Lustre: Skipped 1 previous similar message [14496.496761] Lustre: lustre-MDT0001: new disk, initializing [14496.581658] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14496.631961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14496.653287] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14502.010263] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [14514.003721] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14523.678052] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14531.552370] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14531.686779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14531.701332] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [14532.161310] Lustre: lustre-OST0000: new disk, initializing [14532.175285] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [14532.305883] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14537.183973] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [14540.359970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14540.374783] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14549.138737] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14558.184938] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14558.205146] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14558.231445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14561.252415] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14561.272559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14561.279423] Lustre: Skipped 1 previous similar message [14561.298198] Lustre: Skipped 1 previous similar message [14564.311340] Lustre: server umount lustre-MDT0000 complete [14566.371292] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14566.390746] LustreError: Skipped 2 previous similar messages [14569.243835] LustreError: 479307:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776214739 with bad export cookie 408371482941684667 [14569.247653] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14569.267608] LustreError: 479307:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [14569.791199] Lustre: server umount lustre-MDT0001 complete [14580.704278] Lustre: 482566:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776214745/real 1776214745] req@00000000b5874c4f x1862496006720000/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776214751 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [14581.067822] Lustre: server umount lustre-OST0000 complete [14596.323414] Lustre: DEBUG MARKER: == conf-sanity test 73: failnode to update from mountdata properly ========================================================== 20:59:24 (1776214764) [14597.801895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14609.949771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14611.261523] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14611.370792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14616.545907] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14618.006331] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [14621.675054] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14626.829133] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14627.900160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14628.281192] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14634.374174] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [14645.337541] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14652.843898] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14661.843431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14661.934414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14662.271442] Lustre: Found index 0 for lustre-OST0000, updating log [14662.619544] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14666.961965] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [14677.069979] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14687.232839] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5 to 0x0:33 [14694.370495] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14694.390280] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14694.410739] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14694.416330] Lustre: Skipped 1 previous similar message [14699.691550] Lustre: server umount lustre-OST0000 complete [14704.615158] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14704.623333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14704.635974] Lustre: Skipped 2 previous similar messages [14704.648352] Lustre: Skipped 2 previous similar messages [14709.731334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14709.737022] Lustre: Skipped 1 previous similar message [14710.751588] Lustre: server umount lustre-MDT0000 complete [14715.065307] LustreError: 483376:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776214885 with bad export cookie 408371482941686004 [14715.069687] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14715.078433] LustreError: 483376:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [14728.818723] Lustre: DEBUG MARKER: == conf-sanity test 75: The order of --index should be irrelevant ========================================================== 21:01:36 (1776214896) [14738.024192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14743.897578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14749.902986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14757.190141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14777.831429] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [14790.621996] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [14803.297650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14811.103553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14817.264850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14824.916241] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14833.758160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14833.901703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14834.066785] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [14834.094459] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [14834.188284] Lustre: lustre-MDT0000: new disk, initializing [14834.347616] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14834.410615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14840.530437] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [14851.155174] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14851.197036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14851.251398] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [14851.283896] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [14851.290119] Lustre: Skipped 1 previous similar message [14851.335271] Lustre: lustre-MDT0001: new disk, initializing [14851.469996] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14851.493626] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14857.266663] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [14869.336148] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14877.750162] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14886.409986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14886.601644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14886.615226] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [14886.980806] Lustre: lustre-OST0000: new disk, initializing [14886.989233] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [14887.143360] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14887.153344] Lustre: Skipped 1 previous similar message [14892.064857] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all 8 [14897.331028] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14897.355149] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14906.146041] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14913.492659] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [14913.769548] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [14920.413670] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [14920.659732] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [14923.238068] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14923.243229] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14923.250659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14923.250668] Lustre: Skipped 2 previous similar messages [14923.262798] Lustre: Skipped 1 previous similar message [14923.301211] LustreError: Skipped 1 previous similar message [14929.072137] Lustre: server umount lustre-OST0000 complete [14929.079568] Lustre: Skipped 1 previous similar message [14938.593630] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14938.599423] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14938.620153] Lustre: Skipped 1 previous similar message [14943.731484] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14943.765220] LustreError: Skipped 1 previous similar message [14944.330520] LustreError: 490457:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776215115 with bad export cookie 408371482941687075 [14944.334792] LustreError: 166-1: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14944.347802] LustreError: 490457:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [14945.096304] Lustre: server umount lustre-MDT0001 complete [14945.098319] Lustre: Skipped 1 previous similar message [14977.096864] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [14988.514484] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [15002.612289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [15013.470368] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [15020.215104] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [15026.980334] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [15028.947225] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14783 sec ========= 21:06:37 (1776215197) [15078.913344] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [15082.190383] Key type lgssc unregistered [15082.546677] LNet: 499205:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15082.581490] LNet: Removed LNI 192.168.204.144@tcp [15083.563753] Key type .llcrypt unregistered [15083.565465] Key type ._llcrypt unregistered