[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 536782983 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002000] x2apic enabled [ 0.002000] Switched APIC routing to physical x2apic. [ 0.002000] kvm-guest: setup PV IPIs [ 0.003936] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.004006] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.005018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.006008] pid_max: default: 32768 minimum: 301 [ 0.007000] LSM: Security Framework initializing [ 0.007000] Yama: becoming mindful. [ 0.007029] SELinux: Initializing. [ 0.008050] *** VALIDATE selinux *** [ 0.012434] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.016661] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.017136] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.018095] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.019086] *** VALIDATE tmpfs *** [ 0.020397] *** VALIDATE proc *** [ 0.022150] *** VALIDATE cgroup *** [ 0.023005] *** VALIDATE cgroup2 *** [ 0.024218] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.025123] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.026007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.027023] Spectre V2 : User space: Vulnerable [ 0.028011] Speculative Store Bypass: Vulnerable [ 0.031074] debug: unmapping init [mem 0xffffffffa6a59000-0xffffffffa6a60fff] [ 0.032000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.032571] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.033018] ... version: 2 [ 0.034009] ... bit width: 48 [ 0.035008] ... generic registers: 4 [ 0.036007] ... value mask: 0000ffffffffffff [ 0.037008] ... max period: 00007fffffffffff [ 0.038007] ... fixed-purpose events: 3 [ 0.039007] ... event mask: 000000070000000f [ 0.040247] rcu: Hierarchical SRCU implementation. [ 0.042320] smp: Bringing up secondary CPUs ... [ 0.043469] x86: Booting SMP configuration: [ 0.044013] .... node #0, CPUs: #1 #2 #3 [ 0.061091] smp: Brought up 1 node, 4 CPUs [ 0.063008] smpboot: Max logical packages: 1 [ 0.064009] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.113051] node 0 deferred pages initialised in 43ms [ 0.117468] devtmpfs: initialized [ 0.118248] x86/mm: Memory block size: 128MB [ 0.120819] gcov: version magic: 0x41383552 [ 0.124329] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.125054] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.127252] pinctrl core: initialized pinctrl subsystem [ 0.129155] [ 0.129757] ************************************************************* [ 0.132008] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.135010] ** ** [ 0.137008] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.139008] ** ** [ 0.142008] ** This means that this kernel is built to expose internal ** [ 0.144009] ** IOMMU data structures, which may compromise security on ** [ 0.146009] ** your system. ** [ 0.148008] ** ** [ 0.150007] ** If you see this message and you are not debugging the ** [ 0.152009] ** kernel, report this immediately to your vendor! ** [ 0.154008] ** ** [ 0.155010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.157009] ************************************************************* [ 0.159629] NET: Registered protocol family 16 [ 0.162545] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.164053] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.167086] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.170528] cpuidle: using governor menu [ 0.177551] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.180402] PCI: Using configuration type 1 for base access [ 0.182134] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.191053] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.193012] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.197113] cryptd: max_cpu_qlen set to 1000 [ 0.201230] ACPI: Added _OSI(Module Device) [ 0.202011] ACPI: Added _OSI(Processor Device) [ 0.204009] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.206009] ACPI: Added _OSI(Processor Aggregator Device) [ 0.211670] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.218361] ACPI: Interpreter enabled [ 0.219047] ACPI: PM: (supports S0 S3 S4 S5) [ 0.221008] ACPI: Using IOAPIC for interrupt routing [ 0.222083] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.228464] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.238486] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.241025] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.243011] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.246064] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.249762] acpiphp: Slot [2] registered [ 0.251075] acpiphp: Slot [3] registered [ 0.253090] acpiphp: Slot [4] registered [ 0.254089] acpiphp: Slot [5] registered [ 0.255108] acpiphp: Slot [6] registered [ 0.256000] acpiphp: Slot [7] registered [ 0.256000] acpiphp: Slot [8] registered [ 0.256080] acpiphp: Slot [9] registered [ 0.257000] acpiphp: Slot [10] registered [ 0.258085] acpiphp: Slot [11] registered [ 0.260061] acpiphp: Slot [12] registered [ 0.261056] acpiphp: Slot [13] registered [ 0.262106] acpiphp: Slot [14] registered [ 0.263064] acpiphp: Slot [15] registered [ 0.264067] acpiphp: Slot [16] registered [ 0.265116] acpiphp: Slot [17] registered [ 0.267016] acpiphp: Slot [18] registered [ 0.267915] acpiphp: Slot [19] registered [ 0.268057] acpiphp: Slot [20] registered [ 0.268951] acpiphp: Slot [21] registered [ 0.270051] acpiphp: Slot [22] registered [ 0.270987] acpiphp: Slot [23] registered [ 0.272088] acpiphp: Slot [24] registered [ 0.274086] acpiphp: Slot [25] registered [ 0.276112] acpiphp: Slot [26] registered [ 0.277148] acpiphp: Slot [27] registered [ 0.279073] acpiphp: Slot [28] registered [ 0.280135] acpiphp: Slot [29] registered [ 0.282099] acpiphp: Slot [30] registered [ 0.283083] acpiphp: Slot [31] registered [ 0.285060] PCI host bridge to bus 0000:00 [ 0.286014] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.288016] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.290012] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.292015] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.294017] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.297035] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.298139] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.302119] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.305000] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.316799] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.320488] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.322015] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.324015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.326010] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.329311] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.330713] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.333024] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.334637] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.340012] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.353012] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.355000] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.359598] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.365012] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.371012] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.388018] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.395793] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.400020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.405012] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.422013] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.429233] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.437011] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.445016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.459017] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.465000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.470012] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.476011] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.490012] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.497116] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.506011] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.514012] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.526016] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.533000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.537011] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.545015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.562039] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.568829] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.571376] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.573324] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.575367] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.578165] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.583003] iommu: Default domain type: Passthrough [ 0.584416] SCSI subsystem initialized [ 0.586106] ACPI: bus type USB registered [ 0.587097] usbcore: registered new interface driver usbfs [ 0.589045] usbcore: registered new interface driver hub [ 0.591049] usbcore: registered new device driver usb [ 0.593151] pps_core: LinuxPPS API ver. 1 registered [ 0.594006] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.597034] PTP clock support registered [ 0.600332] EDAC MC: Ver: 3.0.0 [ 0.602109] PCI: Using ACPI for IRQ routing [ 0.603000] NetLabel: Initializing [ 0.603000] NetLabel: domain hash size = 128 [ 0.605007] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.607084] NetLabel: unlabeled traffic allowed by default [ 0.609093] vgaarb: loaded [ 0.610190] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.611000] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.616000] clocksource: Switched to clocksource kvm-clock [ 0.727357] VFS: Disk quotas dquot_6.6.0 [ 0.728571] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.730454] *** VALIDATE ramfs *** [ 0.731286] *** VALIDATE hugetlbfs *** [ 0.732433] pnp: PnP ACPI init [ 0.734321] pnp: PnP ACPI: found 6 devices [ 0.750384] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.753066] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.754715] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.756482] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.758300] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.760082] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.762071] NET: Registered protocol family 2 [ 0.763911] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.768184] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.770767] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.776017] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.779334] TCP: Hash tables configured (established 65536 bind 65536) [ 0.782790] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.785396] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.788149] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.790364] NET: Registered protocol family 1 [ 0.794438] RPC: Registered named UNIX socket transport module. [ 0.796219] RPC: Registered udp transport module. [ 0.798225] RPC: Registered tcp transport module. [ 0.800225] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.803627] NET: Registered protocol family 44 [ 0.804983] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.806616] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.808099] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.809781] PCI: CLS 0 bytes, default 64 [ 0.811109] Unpacking initramfs... [ 2.367294] debug: unmapping init [mem 0xffff9af83cc54000-0xffff9af83ffbffff] [ 2.371039] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.373069] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.375537] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.967757] Initialise system trusted keyrings [ 2.969128] Key type blacklist registered [ 2.972026] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.981122] zbud: loaded [ 2.984965] *** VALIDATE nfs *** [ 2.985741] *** VALIDATE nfs4 *** [ 2.986873] pstore: using deflate compression [ 2.990643] Platform Keyring initialized [ 3.105946] NET: Registered protocol family 38 [ 3.107508] Key type asymmetric registered [ 3.108898] Asymmetric key parser 'x509' registered [ 3.110554] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.113214] io scheduler mq-deadline registered [ 3.114812] io scheduler kyber registered [ 3.116439] io scheduler bfq registered [ 3.118721] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.121247] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.123811] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.126370] ACPI: Power Button [PWRF] [ 3.223306] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.331639] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.540638] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.643616] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.855582] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.885760] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.921315] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.929704] Non-volatile memory driver v1.3 [ 3.931637] Linux agpgart interface v0.103 [ 3.968716] virtio_blk virtio1: [vda] 68256 512-byte logical blocks (34.9 MB/33.3 MiB) [ 3.971413] vda: detected capacity change from 0 to 34947072 [ 3.998549] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.001142] vdb: detected capacity change from 0 to 1073741824 [ 4.013781] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.016734] vdc: detected capacity change from 0 to 2621440000 [ 4.029257] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.033353] vdd: detected capacity change from 0 to 2621440000 [ 4.051950] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.054333] vde: detected capacity change from 0 to 4294967296 [ 4.074816] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.077239] vdf: detected capacity change from 0 to 4294967296 [ 4.084264] libphy: Fixed MDIO Bus: probed [ 4.089515] usbcore: registered new interface driver usbserial_generic [ 4.095186] usbserial: USB Serial support registered for generic [ 4.097332] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.101421] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.103776] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.106437] mousedev: PS/2 mouse device common for all mice [ 4.109205] rtc_cmos 00:05: RTC can wake from S4 [ 4.112211] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.116421] rtc_cmos 00:05: registered as rtc0 [ 4.120151] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.121959] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.128371] intel_pstate: CPU model not supported [ 4.129302] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.136326] hid: raw HID events driver (C) Jiri Kosina [ 4.138265] usbcore: registered new interface driver usbhid [ 4.139841] usbhid: USB HID core driver [ 4.141325] drop_monitor: Initializing network drop monitor service [ 4.143405] Initializing XFRM netlink socket [ 4.145514] NET: Registered protocol family 10 [ 4.148708] Segment Routing with IPv6 [ 4.150252] NET: Registered protocol family 17 [ 4.152518] mpls_gso: MPLS GSO support [ 4.160568] RAS: Correctable Errors collector initialized. [ 4.162758] AVX version of gcm_enc/dec engaged. [ 4.165249] AES CTR mode by8 optimization enabled [ 4.254596] sched_clock: Marking stable (4254578234, 0)->(5220473299, -965895065) [ 4.257423] registered taskstats version 1 [ 4.258542] Loading compiled-in X.509 certificates [ 4.260139] zswap: loaded using pool lzo/zbud [ 4.280331] Key type big_key registered [ 4.291236] Key type encrypted registered [ 4.292797] ima: No TPM chip found, activating TPM-bypass! [ 4.295630] ima: Allocated hash algorithm: sha1 [ 4.296610] ima: No architecture policies found [ 4.297608] evm: Initialising EVM extended attributes: [ 4.298820] evm: security.selinux [ 4.299731] evm: security.ima [ 4.300358] evm: security.capability [ 4.301093] evm: HMAC attrs: 0x1 [ 4.304209] rtc_cmos 00:05: setting system clock to 2025-10-14 14:22:31 UTC (1760451751) [ 4.308895] debug: unmapping init [mem 0xffffffffa7a03000-0xffffffffa7bfffff] [ 4.311041] debug: unmapping init [mem 0xffffffffa6782000-0xffffffffa6a58fff] [ 4.321080] Write protecting the kernel read-only data: 28672k [ 4.324726] debug: unmapping init [mem 0xffffffffa4e03000-0xffffffffa4ffffff] [ 4.327256] debug: unmapping init [mem 0xffffffffa5714000-0xffffffffa57fffff] [ 4.363488] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.371668] systemd[1]: Detected virtualization kvm. [ 4.373885] systemd[1]: Detected architecture x86-64. [ 4.376573] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.401771] systemd[1]: No hostname configured. [ 4.403231] systemd[1]: Set hostname to . [ 4.405229] random: systemd: uninitialized urandom read (16 bytes read) [ 4.408263] systemd[1]: Initializing machine ID from random generator. [ 4.464517] random: ln: uninitialized urandom read (6 bytes read) [ 4.567685] random: systemd: uninitialized urandom read (16 bytes read) [ 4.569679] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 4.574671] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 4.578590] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Swap. [ OK ] Reached target Sockets. [ OK ] Reached target Paths. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.262043] device-mapper: uevent: version 1.0.3 [ 5.263683] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 6.113277] virtio_net virtio0 ens2: renamed from eth0 [ 6.138814] random: fast init done [ 6.265814] scsi host0: ata_piix [ 6.268620] scsi host1: ata_piix [ 6.278368] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 6.280886] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.810192] random: crng init done [ 10.811106] random: 7 urandom warning(s) missed due to ratelimiting [ 11.205879] dracut-initqueue[594]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 11.867257] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 13.288381] printk: systemd: 26 output lines suppressed due to ratelimiting [ 13.565084] SELinux: Disabled at runtime. [ 13.627889] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 13.636839] systemd[1]: Detected virtualization kvm. [ 13.638713] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 14.164260] systemd[1]: initrd-switch-root.service: Succeeded. [ 14.167041] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 14.171144] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 14.175341] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 14.177548] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 14.182786] systemd[1]: Starting Journal Service... Starting Journal Service... [ 14.194794] systemd[1]: Listening on RPCbind Server Activation Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target rpc_pipefs.target. Mounting Kernel Debug File System... Mounting Huge Pages File System... [ OK ] Created slice User and Session Slice. Starting Create list of required st…ce nodes for the current kernel... [ 14.318444] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Switch Root. Starting Apply Kernel Variables... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Mounting POSIX Message Queue File System... [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target Slices. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Stopped target Initrd File Systems. Starting Remount Root and Kernel File Systems... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 14.961011] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 15.419259] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 15.483992] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 16.167298] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 16.177671] EDAC sbridge: Ver: 1.1.2 [ 17.836805] Key type dns_resolver registered [ 18.196536] NFS: Registering the id_resolver key type [ 18.198345] Key type id_resolver registered [ 18.199648] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. Starting Login Service... Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg356-server login: [ 68.879376] libcfs: loading out-of-tree module taints kernel. [ 68.901957] alg: No test for adler32 (adler32-zlib) [ 69.654375] Key type ._llcrypt registered [ 69.656322] Key type .llcrypt registered [ 69.704923] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_hostid [ 77.573709] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 78.273309] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 78.565143] Lustre: Lustre: Build Version: 2.15.7_5_g4a1d30d [ 78.867803] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 78.870890] LNet: Accept secure, port 988 [ 80.496171] Key type lgssc registered [ 81.177701] Lustre: Echo OBD driver; http://www.lustre.org/ [ 89.740885] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 90.952603] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 95.370254] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 98.676566] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 102.191130] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 108.938657] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 114.554513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 114.595154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 115.719035] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 115.733337] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 115.783168] Lustre: lustre-MDT0000: new disk, initializing [ 115.834345] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 115.844414] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 117.616035] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 123.701025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 123.746959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 123.787884] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 123.803433] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 123.806269] Lustre: Skipped 1 previous similar message [ 123.850985] Lustre: lustre-MDT0001: new disk, initializing [ 123.878196] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 123.888681] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 123.892862] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 125.521562] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 131.063781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 131.093567] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 131.097056] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 131.187347] Lustre: lustre-OST0000: new disk, initializing [ 131.189336] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 131.207360] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 132.676856] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 135.671634] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 135.677402] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 139.097603] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 139.141472] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 139.144526] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 139.197690] Lustre: lustre-OST0001: new disk, initializing [ 139.202560] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 139.240169] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 141.445213] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 142.473312] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 142.477475] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 147.944748] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 154.906983] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 162.124190] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing check_logdir /tmp/testlogs/ [ 164.485701] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing yml_node [ 166.875746] Lustre: DEBUG MARKER: Client: 2.15.7.5 [ 168.042343] Lustre: DEBUG MARKER: MDS: 2.15.7.5 [ 169.114104] Lustre: DEBUG MARKER: OSS: 2.15.7.5 [ 169.862304] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Oct 14 10:25:16 EDT 2025 [ 173.450235] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball 110 [ 174.141541] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 [ 188.897177] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 188.897948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 188.901687] Lustre: Skipped 3 previous similar messages [ 188.907065] Lustre: Skipped 3 previous similar messages [ 192.665820] Lustre: server umount lustre-MDT0000 complete [ 194.017132] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 194.023608] LustreError: Skipped 3 previous similar messages [ 194.471213] LustreError: 8230:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760451941 with bad export cookie 8813807300087305740 [ 194.472513] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 194.476230] LustreError: 8230:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 199.136601] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 199.137051] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 199.137537] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 199.142332] LustreError: Skipped 1 previous similar message [ 199.148038] Lustre: Skipped 1 previous similar message [ 200.160828] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 200.160859] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 200.166257] Lustre: Skipped 2 previous similar messages [ 200.179090] LustreError: Skipped 1 previous similar message [ 200.669919] Lustre: server umount lustre-MDT0001 complete [ 202.597462] Lustre: server umount lustre-OST0000 complete [ 204.624898] Lustre: server umount lustre-OST0001 complete [ 207.819287] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_hostid [ 211.889111] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 216.846225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 220.087770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 222.955910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 225.904211] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 229.208647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 229.238410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 229.355122] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 229.371544] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 229.415187] Lustre: lustre-MDT0000: new disk, initializing [ 229.465943] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 229.475305] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 231.430438] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 236.949052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 236.979332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 237.028600] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 237.153729] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 239.187270] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 244.514958] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 247.343516] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 250.818872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 250.854213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 250.858614] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 250.992222] Lustre: lustre-OST0000: new disk, initializing [ 250.994000] Lustre: Skipped 1 previous similar message [ 251.000812] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 251.006426] Lustre: Skipped 2 previous similar messages [ 251.055794] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 251.058748] Lustre: Skipped 1 previous similar message [ 252.304598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 252.308138] Lustre: Skipped 1 previous similar message [ 252.310319] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 253.291624] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 258.827543] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 261.050734] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 261.162293] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 263.185325] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 263.284806] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 267.747575] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 267.751541] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 267.757455] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 270.307704] Lustre: server umount lustre-OST0000 complete [ 272.865614] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 272.868907] LustreError: Skipped 1 previous similar message [ 272.870698] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 272.875092] Lustre: Skipped 1 previous similar message [ 272.879073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 272.881192] Lustre: Skipped 1 previous similar message [ 280.018457] LustreError: 14306:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760452027 with bad export cookie 8813807300087307189 [ 280.021647] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 280.023254] LustreError: 14306:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 280.179085] Lustre: server umount lustre-MDT0001 complete [ 280.180992] Lustre: Skipped 1 previous similar message [ 282.382926] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 10:27:09 (1760452029) [ 283.547120] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 284.417175] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 10:27:11 (1760452031) [ 287.428298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 287.630964] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 287.664767] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 289.609553] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 292.834873] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 293.803549] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 295.924102] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 299.499452] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 302.072586] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 305.371573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 307.350212] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 310.772975] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 325.088963] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 325.092434] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 325.097245] Lustre: Skipped 1 previous similar message [ 325.100686] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 325.106018] Lustre: Skipped 3 previous similar messages [ 329.187078] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 331.036049] Lustre: server umount lustre-OST0000 complete [ 334.305147] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 334.309389] Lustre: Skipped 1 previous similar message [ 339.426238] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 339.432561] LustreError: Skipped 1 previous similar message [ 340.492183] LustreError: 19644:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760452087 with bad export cookie 8813807300087308050 [ 340.495308] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 340.497046] LustreError: 19644:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 346.432648] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 347.602485] Key type lgssc unregistered [ 347.731434] LNet: 22531:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 347.735257] LNet: Removed LNI 192.168.203.156@tcp [ 348.094837] Key type .llcrypt unregistered [ 348.096427] Key type ._llcrypt unregistered [ 359.005043] alg: No test for adler32 (adler32-zlib) [ 359.756397] Key type ._llcrypt registered [ 359.758669] Key type .llcrypt registered [ 359.809708] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 360.328709] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 360.482516] Lustre: Lustre: Build Version: 2.15.7_5_g4a1d30d [ 360.580353] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 360.582849] LNet: Accept secure, port 988 [ 362.176184] Key type lgssc registered [ 362.603318] Lustre: Echo OBD driver; http://www.lustre.org/ [ 366.185619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 367.388614] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 367.411552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 369.513915] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 372.706346] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 373.590568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 373.773587] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 375.715680] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 379.200288] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 382.069214] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 385.121281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 385.307759] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 387.145276] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 390.350430] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 397.608812] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 408.035223] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 408.035914] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 408.037909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 408.040064] Lustre: Skipped 1 previous similar message [ 411.628512] Lustre: server umount lustre-OST0000 complete [ 414.689298] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 414.693205] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 414.699593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 414.701637] Lustre: Skipped 1 previous similar message [ 418.273647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 418.276118] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 419.627648] Lustre: server umount lustre-MDT0000 complete [ 421.629253] LustreError: 23974:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760452168 with bad export cookie 9719218043640408638 [ 421.630296] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 421.638929] LustreError: 23974:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 421.791520] Lustre: server umount lustre-MDT0001 complete [ 428.157561] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 429.402411] Key type lgssc unregistered [ 429.528840] LNet: 27665:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 429.534953] LNet: Removed LNI 192.168.203.156@tcp [ 429.917665] Key type .llcrypt unregistered [ 429.918886] Key type ._llcrypt unregistered [ 441.820517] alg: No test for adler32 (adler32-zlib) [ 442.572398] Key type ._llcrypt registered [ 442.574603] Key type .llcrypt registered [ 442.646604] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 443.086241] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 443.393677] Lustre: Lustre: Build Version: 2.15.7_5_g4a1d30d [ 443.511690] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 443.514277] LNet: Accept secure, port 988 [ 445.120118] Key type lgssc registered [ 445.698942] Lustre: Echo OBD driver; http://www.lustre.org/ [ 449.710676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 450.920704] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 450.952218] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 452.903845] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 456.161579] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 456.686752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 456.860145] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 458.702252] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 461.858214] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 464.169177] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 466.854076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 466.998061] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 468.791306] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 472.066702] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 481.322407] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 486.372414] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 486.375268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 486.382420] Lustre: Skipped 1 previous similar message [ 487.977809] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.56@tcp (stopping) [ 487.982137] Lustre: Skipped 2 previous similar messages [ 491.488880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 491.491628] Lustre: Skipped 2 previous similar messages [ 491.595910] Lustre: server umount lustre-MDT0000 complete [ 493.088852] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 493.145461] LustreError: 29104:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760452240 with bad export cookie 782096880831435414 [ 493.147856] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 493.150309] LustreError: 29104:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 496.609057] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 496.609239] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 496.618661] Lustre: Skipped 1 previous similar message [ 496.620811] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 499.340068] Lustre: server umount lustre-MDT0001 complete [ 517.618304] Lustre: server umount lustre-OST0000 complete [ 524.882636] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 525.978737] Key type lgssc unregistered [ 526.096381] LNet: 33124:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 526.099645] LNet: Removed LNI 192.168.203.156@tcp [ 526.379176] Key type .llcrypt unregistered [ 526.380596] Key type ._llcrypt unregistered [ 535.858300] alg: No test for adler32 (adler32-zlib) [ 536.609580] Key type ._llcrypt registered [ 536.611107] Key type .llcrypt registered [ 536.654872] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 537.109204] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 537.264546] Lustre: Lustre: Build Version: 2.15.7_5_g4a1d30d [ 537.351336] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 537.354183] LNet: Accept secure, port 988 [ 538.944183] Key type lgssc registered [ 539.322185] Lustre: Echo OBD driver; http://www.lustre.org/ [ 542.314457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 543.492184] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 543.514675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 545.121914] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 548.448039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 548.612537] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 550.278168] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 553.008710] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 555.130189] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 557.615576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 557.734414] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 558.758841] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 559.277772] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 562.233976] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 573.994680] Lustre: lustre-OST0000: Not available for connect from 192.168.203.56@tcp (stopping) [ 574.434875] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 574.435854] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 574.438945] LustreError: Skipped 1 previous similar message [ 574.444244] Lustre: Skipped 1 previous similar message [ 575.710348] Lustre: server umount lustre-OST0000 complete [ 589.794093] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 589.794162] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 589.795420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 589.795425] Lustre: Skipped 2 previous similar messages [ 589.798320] Lustre: Skipped 1 previous similar message [ 590.817591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 590.819727] Lustre: Skipped 2 previous similar messages [ 594.759388] Lustre: server umount lustre-MDT0000 complete [ 595.936832] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 595.940708] LustreError: Skipped 1 previous similar message [ 596.033982] LustreError: 34563:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760452343 with bad export cookie 3277807553910602925 [ 596.037022] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 596.039677] LustreError: 34563:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 602.234645] Lustre: server umount lustre-MDT0001 complete [ 607.156598] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 608.266813] Key type lgssc unregistered [ 608.386319] LNet: 38412:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 608.389711] LNet: Removed LNI 192.168.203.156@tcp [ 608.694691] Key type .llcrypt unregistered [ 608.695880] Key type ._llcrypt unregistered [ 618.250991] alg: No test for adler32 (adler32-zlib) [ 619.002459] Key type ._llcrypt registered [ 619.004049] Key type .llcrypt registered [ 619.045118] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 619.474950] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 619.668467] Lustre: Lustre: Build Version: 2.15.7_5_g4a1d30d [ 619.754420] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 619.756793] LNet: Accept secure, port 988 [ 621.344145] Key type lgssc registered [ 621.792814] Lustre: Echo OBD driver; http://www.lustre.org/ [ 624.804861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 625.951179] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 625.968268] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 627.553461] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 630.468447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 630.594682] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 632.074671] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 634.780559] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 636.866269] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 639.273118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 639.392578] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 640.878675] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 641.446874] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 643.642994] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 650.937240] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 652.261889] LNetError: 120-3: Refusing connection from 192.168.203.156 for 127.0.0.2@tcp: No matching NI [ 652.265132] LNetError: 39175:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 652.267861] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 652.362802] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 663.258222] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2025-10-1410h33m30s [ 663.866055] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 665.638907] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.56@tcp (stopping) [ 666.082325] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 666.086886] Lustre: Skipped 2 previous similar messages [ 666.090070] LNetError: 120-3: Refusing connection from 192.168.203.156 for 127.0.0.2@tcp: No matching NI [ 666.092948] LNetError: 39172:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 666.095884] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 670.751452] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.56@tcp (stopping) [ 670.754622] Lustre: Skipped 3 previous similar messages [ 670.791834] Lustre: server umount lustre-MDT0000 complete [ 671.202043] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 671.205405] LNetError: 120-3: Refusing connection from 192.168.203.156 for 127.0.0.2@tcp: No matching NI [ 671.208318] LustreError: Skipped 2 previous similar messages [ 671.212474] LNetError: 39173:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 671.216042] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 672.037260] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 674.505482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 674.555441] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 674.557509] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 674.557673] LustreError: 11-0: MGC192.168.203.156@tcp: operation mgs_target_reg to node 0@lo failed: rc = -107 [ 674.560690] LustreError: Skipped 2 previous similar messages [ 674.566990] Lustre: Evicted from MGS (at 192.168.203.156@tcp) after server handle changed from 0x9d77170ff4639282 to 0x9d77170ff46396cd [ 674.574470] Lustre: MGC192.168.203.156@tcp: Connection restored to 192.168.203.156@tcp (at 0@lo) [ 674.667474] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 674.703716] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:193 [ 676.439390] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 677.813100] Lustre: DEBUG MARKER: Wait for df (28848) ... [ 678.388678] Lustre: DEBUG MARKER: done [ 679.907905] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 679.908466] LNetError: 120-3: Refusing connection from 192.168.203.156 for 127.0.0.2@tcp: No matching NI [ 679.908862] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 679.909803] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 679.910322] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to (at 0@lo) [ 679.911210] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 679.915166] LNetError: 39174:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 679.918889] Lustre: Skipped 1 previous similar message [ 679.924728] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 685.026849] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 685.028965] Lustre: Skipped 1 previous similar message [ 685.592829] Lustre: server umount lustre-OST0000 complete [ 690.145221] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 690.145407] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 690.148364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 690.153650] Lustre: Skipped 2 previous similar messages [ 690.159054] LNetError: 120-3: Refusing connection from 192.168.203.156 for 127.0.0.2@tcp: No matching NI [ 690.162108] LNetError: 39175:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 690.165852] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 700.385037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 700.387230] Lustre: Skipped 4 previous similar messages [ 701.408104] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 701.446113] Lustre: server umount lustre-MDT0000 complete [ 702.755066] LustreError: 41823:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760452449 with bad export cookie 11346563143475369677 [ 702.758345] LustreError: 41823:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 702.758506] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 707.659306] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 708.686195] Key type lgssc unregistered [ 708.809384] LNet: 44966:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 708.811468] LNet: Removed LNI 192.168.203.156@tcp [ 709.104261] Key type .llcrypt unregistered [ 709.105537] Key type ._llcrypt unregistered [ 720.364927] alg: No test for adler32 (adler32-zlib) [ 721.116339] Key type ._llcrypt registered [ 721.117336] Key type .llcrypt registered [ 721.152683] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_hostid [ 727.226312] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 727.680751] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 727.812071] Lustre: Lustre: Build Version: 2.15.7_5_g4a1d30d [ 727.893823] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 727.896333] LNet: Accept secure, port 988 [ 729.480114] Key type lgssc registered [ 729.846701] Lustre: Echo OBD driver; http://www.lustre.org/ [ 732.999930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 735.373387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 737.492271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 739.728246] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 742.079693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 742.102352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 743.186540] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 743.198351] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 743.235799] Lustre: lustre-MDT0000: new disk, initializing [ 743.262305] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 743.268935] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 744.777021] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 748.975800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 748.999750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 749.029887] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 749.045696] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 749.047311] Lustre: Skipped 1 previous similar message [ 749.082881] Lustre: lustre-MDT0001: new disk, initializing [ 749.113273] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 749.125857] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 749.128831] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 750.553535] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 754.488813] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 756.448232] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 758.729205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 758.759486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 758.762327] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 758.852441] Lustre: lustre-OST0000: new disk, initializing [ 758.854775] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 758.878254] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 760.297658] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 764.118733] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 765.609697] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 767.987240] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 767.992485] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 768.743857] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 770.025900] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 770.098822] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 773.089861] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 773.090037] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 773.092631] LustreError: Skipped 1 previous similar message [ 773.093166] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 773.095616] Lustre: Skipped 1 previous similar message [ 773.101215] Lustre: Skipped 1 previous similar message [ 776.858259] Lustre: server umount lustre-OST0000 complete [ 778.209579] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 778.210156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 778.212719] Lustre: Skipped 1 previous similar message [ 783.328865] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 783.330684] Lustre: Skipped 2 previous similar messages [ 784.321200] Lustre: server umount lustre-MDT0000 complete [ 785.571871] LustreError: 49049:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760452532 with bad export cookie 9885801403499558808 [ 785.574316] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 785.575930] LustreError: 49049:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 785.680675] Lustre: server umount lustre-MDT0001 complete [ 788.310987] Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 10:35:35 (1760452535) [ 790.683187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 790.847339] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 790.866180] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 792.250453] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 794.986693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 795.092078] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 796.490944] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 798.849953] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 800.775225] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 803.086971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 803.191066] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 804.593843] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 807.111726] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 814.407561] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2025-10-1410h36m01s [ 814.962946] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 817.163880] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 817.944618] Lustre: DEBUG MARKER: Creating a test file and stat it [ 817.958518] LustreError: 56420:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 136 sleeping for 40000ms [ 823.268316] LNetError: 120-3: Refusing connection from 192.168.203.156 for 127.0.0.2@tcp: No matching NI [ 823.271316] LNetError: 47272:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 823.274406] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 838.190712] Lustre: lustre-MDT0000: Client e7670803-eda9-460e-8b23-545864cef1f3 (at 192.168.203.56@tcp) reconnecting [ 858.000101] LustreError: 56420:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 136 awake [ 858.695668] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 859.486675] Lustre: DEBUG MARKER: done [ 861.666458] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 861.666496] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 861.667080] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 861.670345] Lustre: Skipped 1 previous similar message [ 866.785727] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 866.787263] Lustre: Skipped 2 previous similar messages [ 867.284252] Lustre: server umount lustre-OST0000 complete [ 871.905135] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 871.905497] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 871.911166] Lustre: Skipped 1 previous similar message [ 871.913748] LNetError: 120-3: Refusing connection from 192.168.203.156 for 127.0.0.2@tcp: No matching NI [ 871.916807] LNetError: 47269:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 871.918586] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 874.626776] Lustre: server umount lustre-MDT0000 complete [ 875.881891] LustreError: 56416:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760452623 with bad export cookie 9885801403499559648 [ 875.883879] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 875.884641] LustreError: 56416:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 880.559750] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 881.503308] Key type lgssc unregistered [ 881.608305] LNet: 58034:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 881.610851] LNet: Removed LNI 192.168.203.156@tcp [ 881.874691] Key type .llcrypt unregistered [ 881.876042] Key type ._llcrypt unregistered [ 892.421557] alg: No test for adler32 (adler32-zlib) [ 893.173384] Key type ._llcrypt registered [ 893.174951] Key type .llcrypt registered [ 893.210598] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_hostid [ 898.715604] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 899.014623] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 899.223526] Lustre: Lustre: Build Version: 2.15.7_5_g4a1d30d [ 899.292140] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 899.293820] LNet: Accept secure, port 988 [ 900.872107] Key type lgssc registered [ 901.174362] Lustre: Echo OBD driver; http://www.lustre.org/ [ 903.849965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 905.860445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 907.675495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 909.678615] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 912.023076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 912.046300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 913.128921] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 913.139842] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 913.177305] Lustre: lustre-MDT0000: new disk, initializing [ 913.200812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 913.207642] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 914.718265] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 918.974927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 918.996362] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 919.024916] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 919.041172] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 919.042832] Lustre: Skipped 1 previous similar message [ 919.079210] Lustre: lustre-MDT0001: new disk, initializing [ 919.111550] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 919.124526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 919.128451] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 920.560565] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 924.426730] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 926.360477] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 928.710950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 928.737897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 928.740727] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 928.825162] Lustre: lustre-OST0000: new disk, initializing [ 928.826632] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 928.851441] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 930.207325] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 933.785536] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 935.102779] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 935.923226] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 935.925702] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 936.189031] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 937.537225] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 937.608338] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 941.027428] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 941.028234] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 941.031434] LustreError: Skipped 1 previous similar message [ 941.037477] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 941.040816] Lustre: Skipped 1 previous similar message [ 944.335734] Lustre: server umount lustre-OST0000 complete [ 946.145224] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 946.145880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 946.148542] Lustre: Skipped 2 previous similar messages [ 951.264798] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 951.267267] Lustre: Skipped 2 previous similar messages [ 951.679255] Lustre: server umount lustre-MDT0000 complete [ 952.942958] LustreError: 62976:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760452700 with bad export cookie 5453554114307958545 [ 952.945252] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 952.946201] LustreError: 62976:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 953.060320] Lustre: server umount lustre-MDT0001 complete [ 955.769105] Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 10:38:23 (1760452703) [ 956.237952] Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST [ 956.780427] Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 10:38:24 (1760452704) [ 960.295306] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 961.627108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 961.752808] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 961.767402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 962.100160] Lustre: Failing over lustre-MDT0000 [ 962.102411] LustreError: 67144:0:(osp_object.c:629:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 962.107563] LustreError: 67144:0:(lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 962.111281] LustreError: 67144:0:(lod_dev.c:424:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000 get update log failed: rc = -5 [ 968.672094] Lustre: 67195:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760452709/real 1760452709] req@000000000877b553 x1845968393749312/t0(0) o251->MGC192.168.203.156@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1760452715 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 968.725043] Lustre: server umount lustre-MDT0000 complete [ 970.848972] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 10:38:38 (1760452718) [ 972.998742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 973.134386] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 973.148149] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 974.609905] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 977.378880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 978.890807] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 981.444728] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 983.382746] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 985.677308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 985.823137] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 985.826165] Lustre: Skipped 1 previous similar message [ 987.373125] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 989.885419] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 996.916249] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1002.466308] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1002.469136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1002.469881] Lustre: Skipped 2 previous similar messages [ 1002.472992] Lustre: Skipped 1 previous similar message [ 1004.738943] Lustre: server umount lustre-MDT0000 complete [ 1005.947770] LustreError: 67746:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760452753 with bad export cookie 5453554114307959672 [ 1005.949991] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1005.951836] LustreError: 67746:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1007.174300] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1008.665802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1011.397029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1012.641275] Lustre: Evicted from MGS (at 192.168.203.156@tcp) after server handle changed from 0x4baeea92a2b22b78 to 0x4baeea92a2b232fd [ 1012.645500] Lustre: MGC192.168.203.156@tcp: Connection restored to 192.168.203.156@tcp (at 0@lo) [ 1012.741227] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1012.763582] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1014.116381] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 1014.817114] Lustre: 60351:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760452754/real 1760452754] req@00000000af461333 x1845968393763648/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1760452761 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1014.823322] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1014.826802] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1014.829802] LustreError: Skipped 1 previous similar message [ 1016.847182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1016.864077] Lustre: 60348:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760452757/real 1760452757] req@000000008676ee68 x1845968393763712/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1760452764 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1016.983867] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:65 [ 1018.021825] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 1018.369919] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 1019.040112] Lustre: 60349:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760452759/real 1760452759] req@000000009a6dbfa1 x1845968393763776/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1760452766 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1020.981723] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1022.949560] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1027.553140] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1027.555436] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1027.559432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1030.913382] Lustre: server umount lustre-MDT0000 complete [ 1030.915278] Lustre: Skipped 1 previous similar message [ 1032.185065] LustreError: 67745:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760452779 with bad export cookie 5453554114307961597 [ 1032.187411] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1032.189026] LustreError: 67745:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1034.442479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1037.421884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1038.753232] Lustre: Evicted from MGS (at 192.168.203.156@tcp) after server handle changed from 0x4baeea92a2b232fd to 0x4baeea92a2b23901 [ 1038.756626] Lustre: MGC192.168.203.156@tcp: Connection restored to (at 0@lo) [ 1038.758135] Lustre: Skipped 1 previous similar message [ 1038.849359] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1038.889730] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 1039.904138] Lustre: 60351:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760452780/real 1760452780] req@00000000a4ef6c54 x1845968393774784/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1760452787 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1039.913246] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1039.918734] Lustre: Skipped 2 previous similar messages [ 1040.200368] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 1042.923873] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1043.067680] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:97 [ 1044.070569] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 1044.488629] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 1047.095456] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1049.004327] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1053.152955] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1053.155150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1053.156698] Lustre: Skipped 3 previous similar messages [ 1056.834683] Lustre: server umount lustre-MDT0000 complete [ 1056.835970] Lustre: Skipped 1 previous similar message [ 1058.044313] LustreError: 67745:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760452805 with bad export cookie 5453554114307963137 [ 1058.046456] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1058.048390] LustreError: 67745:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1060.173684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1061.309832] Lustre: DEBUG MARKER: files compared the same [ 1068.000149] Lustre: 76884:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760452809/real 1760452809] req@00000000bb595c61 x1845968393784448/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1760452815 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1068.006563] Lustre: 76884:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 1074.502873] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 1075.560740] Key type lgssc unregistered [ 1075.672331] LNet: 77789:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1075.674768] LNet: Removed LNI 192.168.203.156@tcp [ 1075.967189] Key type .llcrypt unregistered [ 1075.968508] Key type ._llcrypt unregistered [ 1084.434293] alg: No test for adler32 (adler32-zlib) [ 1085.185422] Key type ._llcrypt registered [ 1085.186715] Key type .llcrypt registered [ 1085.219993] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 1085.500454] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1085.709466] Lustre: Lustre: Build Version: 2.15.7_5_g4a1d30d [ 1085.782365] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 1085.785911] LNet: Accept secure, port 988 [ 1087.368114] Key type lgssc registered [ 1087.669050] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1090.025872] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1091.155836] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1091.171148] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1092.478625] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug +malloc all 8 [ 1095.139549] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1095.220935] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1096.512206] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug +malloc all 8 [ 1098.817596] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1100.545038] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1102.655821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1102.742243] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1104.062883] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug +malloc all 8 [ 1105.828593] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 1106.385645] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1109.922573] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:129 [ 1115.104695] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1115.104849] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1115.105541] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1115.108120] Lustre: Skipped 1 previous similar message [ 1119.590638] Lustre: server umount lustre-OST0000 complete [ 1125.344874] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1125.345283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1125.347687] Lustre: Skipped 1 previous similar message [ 1125.352279] Lustre: Skipped 2 previous similar messages [ 1130.464575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1130.466943] Lustre: Skipped 1 previous similar message [ 1135.072101] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1135.112535] Lustre: server umount lustre-MDT0000 complete [ 1136.416422] LustreError: 79231:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760452883 with bad export cookie 15692632691783881351 [ 1136.417772] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1136.419795] LustreError: 79231:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1136.529210] Lustre: server umount lustre-MDT0001 complete [ 1141.151613] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 1142.175080] Key type lgssc unregistered [ 1142.285335] LNet: 82902:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1142.288519] LNet: Removed LNI 192.168.203.156@tcp [ 1142.603247] Key type .llcrypt unregistered [ 1142.604574] Key type ._llcrypt unregistered [ 1151.279440] alg: No test for adler32 (adler32-zlib) [ 1152.030429] Key type ._llcrypt registered [ 1152.032055] Key type .llcrypt registered [ 1152.073557] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 1152.422820] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1152.684464] Lustre: Lustre: Build Version: 2.15.7_5_g4a1d30d [ 1152.774651] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 1152.777262] LNet: Accept secure, port 988 [ 1154.360165] Key type lgssc registered [ 1154.698390] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1157.292534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1174.752261] LustreError: 84334:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.203.156@tcp: failed processing log, type 1: rc = -5 [ 1205.472179] LustreError: 84334:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.203.156@tcp: failed processing log, type 4: rc = -110 [ 1235.169063] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1236.678835] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 1239.364481] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1242.066894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1242.102847] Lustre: *** cfs_fail_loc=706, val=0*** [ 1257.726956] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1257.747273] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1257.761909] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:161 [ 1259.025184] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 1261.678722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1261.785400] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1261.838346] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:161 [ 1263.159508] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 1265.515611] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1267.355900] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1268.192874] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1268.195453] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1268.198961] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1272.290527] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1272.291552] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1272.291575] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1272.294322] Lustre: Skipped 1 previous similar message [ 1274.066329] Lustre: server umount lustre-OST0000 complete [ 1277.409287] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1277.409539] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1277.409900] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1277.416668] Lustre: Skipped 1 previous similar message [ 1281.344021] Lustre: server umount lustre-MDT0000 complete [ 1282.505997] LustreError: 87270:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760453029 with bad export cookie 364453598935290583 [ 1282.508639] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1282.509491] LustreError: 87270:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1282.615501] Lustre: server umount lustre-MDT0001 complete [ 1285.964372] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 1286.891606] Key type lgssc unregistered [ 1287.005442] LNet: 88073:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1287.007801] LNet: Removed LNI 192.168.203.156@tcp [ 1287.263601] Key type .llcrypt unregistered [ 1287.264737] Key type ._llcrypt unregistered [ 1295.705964] alg: No test for adler32 (adler32-zlib) [ 1296.458422] Key type ._llcrypt registered [ 1296.459822] Key type .llcrypt registered [ 1296.500519] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 1296.805700] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1297.008911] Lustre: Lustre: Build Version: 2.15.7_5_g4a1d30d [ 1297.086211] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 1297.088733] LNet: Accept secure, port 988 [ 1298.664111] Key type lgssc registered [ 1299.001962] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1301.509748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1303.797324] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 1306.107945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1306.230766] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1306.257703] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1307.565679] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 1310.187336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1310.278824] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1310.284551] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1310.284822] LustreError: Skipped 3 previous similar messages [ 1311.328773] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1311.332688] LustreError: Skipped 2 previous similar messages [ 1311.332857] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:193 [ 1311.639497] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 1314.450855] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1314.455473] LustreError: Skipped 2 previous similar messages [ 1314.526895] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1314.547739] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:193 [ 1315.891255] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 1325.027340] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1325.028809] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1325.031074] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1325.033195] Lustre: Skipped 1 previous similar message [ 1325.036364] Lustre: Skipped 1 previous similar message [ 1330.063642] Lustre: server umount lustre-OST0000 complete [ 1335.265236] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1335.265267] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1335.265691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1335.272385] Lustre: Skipped 1 previous similar message [ 1337.341815] Lustre: server umount lustre-MDT0000 complete [ 1338.699302] Lustre: server umount lustre-MDT0001 complete [ 1346.016119] Lustre: 92938:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760453087/real 1760453087] req@00000000efb37df2 x1845968811076160/t0(0) o251->MGC192.168.203.156@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1760453093 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1346.028354] Lustre: server umount MGS complete [ 1349.485835] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 10:44:56 (1760453096) [ 1358.798553] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_hostid [ 1361.505133] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 1364.573113] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1366.665759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1368.615381] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1370.630462] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1372.775461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1372.801795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1374.189384] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 1377.915782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1377.936995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1378.011440] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1378.027276] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1378.064241] Lustre: lustre-MDT0001: new disk, initializing [ 1378.086100] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1378.094268] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1378.096122] LustreError: 97630:0:(lod_dev.c:424:lod_sub_recovery_thread()) lustre-MDT0001-osd get update log failed: rc = -115 [ 1379.541333] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 1383.579447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1383.604585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1383.607098] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1383.692424] Lustre: lustre-OST0000: new disk, initializing [ 1383.694133] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1383.713759] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1385.117465] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 1388.839595] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1391.009717] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1391.024147] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1391.058424] Lustre: lustre-MDT0000: new disk, initializing [ 1391.105586] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1391.584575] Lustre: 99373:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@0000000085ecb316 x1845968811083584/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1760453150 ref 2 fl Rpc:RQU/2/0 rc 0/-115 job:'' [ 1392.160542] Lustre: 99483:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@00000000f7d23e1b x1845968811083648/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1760453151 ref 2 fl Rpc:RQU/2/0 rc 0/-115 job:'' [ 1392.397377] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 1397.216488] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1400.674432] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1400.676579] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1401.249854] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 1421.793381] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1421.796680] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1421.802637] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1421.804520] Lustre: Skipped 1 previous similar message [ 1424.355275] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1424.355457] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1424.361618] Lustre: Skipped 1 previous similar message [ 1427.542610] Lustre: server umount lustre-OST0000 complete [ 1429.473600] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1429.473647] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1429.474150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1429.479149] Lustre: Skipped 1 previous similar message [ 1439.712920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1439.716106] Lustre: Skipped 4 previous similar messages [ 1443.296082] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1443.327039] Lustre: server umount lustre-MDT0000 complete [ 1458.144146] Lustre: 100676:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760453199/real 1760453199] req@000000002cbbc352 x1845968811094272/t0(0) o251->MGC192.168.203.156@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1760453205 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1461.875734] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 10:46:49 (1760453209) [ 1468.815440] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 1469.830211] Key type lgssc unregistered [ 1469.938280] LNet: 102081:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1469.940704] LNet: Removed LNI 192.168.203.156@tcp [ 1470.240581] Key type .llcrypt unregistered [ 1470.241778] Key type ._llcrypt unregistered [ 1476.878194] alg: No test for adler32 (adler32-zlib) [ 1477.630359] Key type ._llcrypt registered [ 1477.631568] Key type .llcrypt registered [ 1477.673955] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 1478.204043] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1478.366760] Lustre: Lustre: Build Version: 2.15.7_5_g4a1d30d [ 1478.447555] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 1478.450394] LNet: Accept secure, port 988 [ 1480.040155] Key type lgssc registered [ 1480.423906] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1483.097216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1484.233912] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1484.248459] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1485.629108] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 1488.408721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1488.524430] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1490.011863] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 1492.623870] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1494.701511] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1497.040832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1497.148135] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1498.685544] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 1499.238471] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 1501.224319] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1509.856922] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1509.860140] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1509.865954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1513.954403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1513.954417] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1513.958819] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1515.033832] Lustre: server umount lustre-OST0000 complete [ 1519.073712] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1519.074324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1519.078959] Lustre: Skipped 1 previous similar message [ 1519.082913] Lustre: Skipped 1 previous similar message [ 1522.438806] Lustre: server umount lustre-MDT0000 complete [ 1523.695209] LustreError: 103325:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760453270 with bad export cookie 16462026863096044387 [ 1523.697060] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1523.701174] LustreError: 103325:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1523.813711] Lustre: server umount lustre-MDT0001 complete [ 1528.739083] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 1529.817545] Key type lgssc unregistered [ 1529.928284] LNet: 106997:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1529.932544] LNet: Removed LNI 192.168.203.156@tcp [ 1530.233959] Key type .llcrypt unregistered [ 1530.235580] Key type ._llcrypt unregistered [ 1536.508054] alg: No test for adler32 (adler32-zlib) [ 1537.260404] Key type ._llcrypt registered [ 1537.262119] Key type .llcrypt registered [ 1537.307705] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 1537.841251] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1538.020523] Lustre: Lustre: Build Version: 2.15.7_5_g4a1d30d [ 1538.113689] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 1538.116040] LNet: Accept secure, port 988 [ 1539.704131] Key type lgssc registered [ 1540.067547] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1542.168391] LustreError: 108124:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 716 sleeping [ 1542.170333] LustreError: 108125:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 716 waking [ 1542.174399] LustreError: 108124:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 716 awake: rc=4996 [ 1542.176165] LustreError: 108124:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1542.179975] LustreError: 108124:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1542.180444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1543.329250] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1543.343879] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1545.906909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1545.999601] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1547.423954] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 1549.160229] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing lsmod [ 1550.182579] LustreError: 109433:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 716 sleeping [ 1550.183223] LustreError: 109432:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 716 waking [ 1550.186247] LustreError: 109433:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 716 awake: rc=4998 [ 1550.187935] LustreError: 109433:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1550.190677] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1550.192881] LustreError: 109433:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1550.303592] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1551.306269] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1551.307652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1551.309767] Lustre: Skipped 1 previous similar message [ 1556.452254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1556.453809] Lustre: Skipped 4 previous similar messages [ 1557.380832] Lustre: server umount lustre-MDT0000 complete [ 1558.625993] LustreError: 108142:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760453305 with bad export cookie 18445382457723186699 [ 1558.628548] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1558.631165] LustreError: 108142:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1558.735661] Lustre: server umount lustre-MDT0001 complete [ 1566.176153] Lustre: 110025:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760453307/real 1760453307] req@000000001b9ccba1 x1845969063780608/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1760453313 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1566.250929] Lustre: server umount lustre-OST0000 complete [ 1569.087608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1569.220420] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1569.238712] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1570.637924] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 1573.275038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1573.368252] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1574.704346] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 1577.046149] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1578.881825] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1580.997429] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1582.638205] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 1584.165616] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:65 [ 1585.370242] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1593.313967] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1593.315266] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1593.318017] Lustre: Skipped 2 previous similar messages [ 1593.323041] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1598.434368] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1598.436974] Lustre: Skipped 2 previous similar messages [ 1599.061415] Lustre: server umount lustre-OST0000 complete [ 1603.553499] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1603.553936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1603.556323] Lustre: Skipped 1 previous similar message [ 1606.409461] Lustre: server umount lustre-MDT0000 complete [ 1607.647852] LustreError: 113352:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760453354 with bad export cookie 18445382457723187385 [ 1607.650845] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1607.651121] LustreError: 113352:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1612.740614] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 1613.889873] Key type lgssc unregistered [ 1614.011334] LNet: 114154:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1614.013916] LNet: Removed LNI 192.168.203.156@tcp [ 1614.299479] Key type .llcrypt unregistered [ 1614.300690] Key type ._llcrypt unregistered [ 1623.248791] alg: No test for adler32 (adler32-zlib) [ 1624.000423] Key type ._llcrypt registered [ 1624.002112] Key type .llcrypt registered [ 1624.038882] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 1624.380385] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1624.623191] Lustre: Lustre: Build Version: 2.15.7_5_g4a1d30d [ 1624.712523] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 1624.714439] LNet: Accept secure, port 988 [ 1626.296096] Key type lgssc registered [ 1626.653381] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1629.289611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1630.427223] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1630.441869] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1631.872609] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 1634.622053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1634.733227] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1636.098162] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 1638.619546] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1640.664030] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1642.903437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1643.011730] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1644.489575] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 1645.094121] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 1647.179729] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1654.227181] Lustre: Setting parameter lustre-client.llite.some_wrong_param in log lustre-client [ 1655.044433] Lustre: Modifying parameter lustre-client.llite.some_wrong_param in log lustre-client [ 1659.361280] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1659.362417] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1659.363476] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1659.365627] Lustre: Skipped 1 previous similar message [ 1659.367885] Lustre: Skipped 1 previous similar message [ 1662.037082] Lustre: server umount lustre-OST0000 complete [ 1664.481546] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1664.482248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1664.485655] Lustre: Skipped 1 previous similar message [ 1669.507879] Lustre: server umount lustre-MDT0000 complete [ 1669.601809] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1669.605808] LustreError: Skipped 1 previous similar message [ 1670.810422] LustreError: 115596:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760453418 with bad export cookie 2946199292682193128 [ 1670.813307] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1670.815640] LustreError: 115596:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1670.945041] Lustre: server umount lustre-MDT0001 complete [ 1675.739208] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 1676.699823] Key type lgssc unregistered [ 1676.815479] LNet: 119371:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1676.817939] LNet: Removed LNI 192.168.203.156@tcp [ 1677.134982] Key type .llcrypt unregistered [ 1677.136063] Key type ._llcrypt unregistered [ 1683.929789] alg: No test for adler32 (adler32-zlib) [ 1684.681357] Key type ._llcrypt registered [ 1684.682667] Key type .llcrypt registered [ 1684.721283] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 1685.127742] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1685.312674] Lustre: Lustre: Build Version: 2.15.7_5_g4a1d30d [ 1685.397796] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 1685.399216] LNet: Accept secure, port 988 [ 1686.984121] Key type lgssc registered [ 1687.361405] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1690.105586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1691.240925] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1691.254529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1692.734382] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 1695.690675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1695.815526] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1697.196638] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 1699.679877] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1701.495446] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1703.746363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1703.863339] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1705.405087] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 1706.921672] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 1707.946554] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1717.218475] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1717.219609] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1717.219810] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1717.222532] Lustre: Skipped 1 previous similar message [ 1721.384212] Lustre: server umount lustre-OST0000 complete [ 1726.432872] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1726.436368] LustreError: Skipped 1 previous similar message [ 1726.437557] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1726.441461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1726.444088] Lustre: Skipped 1 previous similar message [ 1727.457375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1727.457885] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1732.577041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1732.578993] Lustre: Skipped 2 previous similar messages [ 1737.184148] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1737.237728] Lustre: server umount lustre-MDT0000 complete [ 1738.587342] LustreError: 120617:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760453485 with bad export cookie 8354171283187547371 [ 1738.590947] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1738.591901] LustreError: 120617:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1738.735542] Lustre: server umount lustre-MDT0001 complete [ 1743.569261] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 1744.670965] Key type lgssc unregistered [ 1744.794438] LNet: 124289:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1744.798532] LNet: Removed LNI 192.168.203.156@tcp [ 1745.139551] Key type .llcrypt unregistered [ 1745.141073] Key type ._llcrypt unregistered [ 1754.310881] alg: No test for adler32 (adler32-zlib) [ 1755.063132] Key type ._llcrypt registered [ 1755.065416] Key type .llcrypt registered [ 1755.101552] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 1755.589904] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1755.770569] Lustre: Lustre: Build Version: 2.15.7_5_g4a1d30d [ 1755.856120] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 1755.858168] LNet: Accept secure, port 988 [ 1757.432097] Key type lgssc registered [ 1757.778160] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1760.396352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1761.524486] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1761.539721] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1763.008879] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 1765.929914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1766.064639] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1767.561695] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 1770.246523] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1772.275921] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1774.574076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1774.691099] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1776.152458] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 1777.766896] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 1778.829802] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1786.212663] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash in log lustre-MDT0000 [ 1787.257393] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 1787.385995] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids in log lustre-MDT0000 [ 1787.388279] Lustre: Skipped 2 previous similar messages [ 1788.718715] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash in log lustre-MDT0000 [ 1788.720130] Lustre: Skipped 2 previous similar messages [ 1793.527667] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 1793.539166] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1793.541276] Lustre: Skipped 1 previous similar message [ 1796.066046] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids in log lustre-MDT0000 [ 1796.068819] Lustre: Skipped 2 previous similar messages [ 1798.972463] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.56@tcp 192.168.0.[2,10]@tcp [ 1801.697593] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1801.698154] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1801.699984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1801.702660] Lustre: Skipped 1 previous similar message [ 1801.705789] Lustre: Skipped 1 previous similar message [ 1806.817276] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1806.819201] Lustre: Skipped 1 previous similar message [ 1807.314304] Lustre: server umount lustre-OST0000 complete [ 1811.937398] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1811.938020] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1811.941645] Lustre: Skipped 1 previous similar message [ 1814.593856] Lustre: server umount lustre-MDT0000 complete [ 1815.781782] LustreError: 126535:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760453562 with bad export cookie 5980780659204124811 [ 1815.784132] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1815.786180] LustreError: 126535:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1815.888440] Lustre: server umount lustre-MDT0001 complete [ 1820.259994] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 1821.168515] Key type lgssc unregistered [ 1821.283449] LNet: 130559:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1821.286433] LNet: Removed LNI 192.168.203.156@tcp [ 1821.564542] Key type .llcrypt unregistered [ 1821.565909] Key type ._llcrypt unregistered [ 1831.102079] alg: No test for adler32 (adler32-zlib) [ 1831.853407] Key type ._llcrypt registered [ 1831.854438] Key type .llcrypt registered [ 1831.897582] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 1832.262651] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1832.562257] Lustre: Lustre: Build Version: 2.15.7_5_g4a1d30d [ 1832.644919] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 1832.646812] LNet: Accept secure, port 988 [ 1834.216158] Key type lgssc registered [ 1834.544585] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1837.081636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1838.202811] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1838.212611] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 1838.216972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1838.220838] Lustre: lustre-MDT0000: nosquash_nids set to 2@gni 0@lo 192.168.203.56@tcp 192.168.0.[2,10]@tcp [ 1839.473781] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 1842.086740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1842.179217] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1842.187211] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.56@tcp 192.168.0.[2,10]@tcp [ 1842.200564] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1843.463305] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 1845.747512] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1847.512349] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1849.655916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1849.750645] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1851.103855] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 1852.837754] Lustre: lustre-OST0000: deleting orphan objects from 0x0:167 to 0x0:193 [ 1853.452487] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1862.114158] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1862.115812] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1862.119265] Lustre: Skipped 1 previous similar message [ 1862.123124] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1867.166398] Lustre: server umount lustre-OST0000 complete [ 1872.354077] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1872.354822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1872.358748] Lustre: Skipped 1 previous similar message [ 1872.361804] Lustre: Skipped 2 previous similar messages [ 1874.628659] Lustre: server umount lustre-MDT0000 complete [ 1875.869894] LustreError: 132142:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760453623 with bad export cookie 18326452866073029068 [ 1875.872411] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1875.873517] LustreError: 132142:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1875.996828] Lustre: server umount lustre-MDT0001 complete [ 1880.901390] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 1882.079402] Key type lgssc unregistered [ 1882.216316] LNet: 135864:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1882.218796] LNet: Removed LNI 192.168.203.156@tcp [ 1882.524204] Key type .llcrypt unregistered [ 1882.525387] Key type ._llcrypt unregistered [ 1891.895775] alg: No test for adler32 (adler32-zlib) [ 1892.647260] Key type ._llcrypt registered [ 1892.648166] Key type .llcrypt registered [ 1892.684056] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_hostid [ 1897.597601] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 1898.073308] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1898.195739] Lustre: Lustre: Build Version: 2.15.7_5_g4a1d30d [ 1898.274289] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 1898.276097] LNet: Accept secure, port 988 [ 1899.848096] Key type lgssc registered [ 1900.206912] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1903.191226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1905.256319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1907.202546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1909.212364] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1911.554297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1911.577954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1912.659601] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1912.669191] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1912.709913] Lustre: lustre-MDT0000: new disk, initializing [ 1912.735104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1912.742573] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1914.131858] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 1918.238575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1918.262579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1918.296189] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1918.311167] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1918.312787] Lustre: Skipped 1 previous similar message [ 1918.346574] Lustre: lustre-MDT0001: new disk, initializing [ 1918.376051] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1918.387337] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1918.391754] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1919.821374] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 1923.797220] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1925.762194] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1928.214481] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1928.241409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1928.245925] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1928.338806] Lustre: lustre-OST0000: new disk, initializing [ 1928.341267] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1928.369295] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1928.812048] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1928.815653] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1929.788991] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 1933.756506] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1935.285670] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1935.365143] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1936.922577] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 1937.016074] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1939.425435] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1939.426181] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1939.428188] LustreError: Skipped 1 previous similar message [ 1939.428743] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1939.432365] Lustre: Skipped 1 previous similar message [ 1943.830395] Lustre: server umount lustre-OST0000 complete [ 1949.152820] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1949.155502] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1949.161387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1949.163247] Lustre: Skipped 1 previous similar message [ 1954.785277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1954.787562] Lustre: Skipped 2 previous similar messages [ 1959.392137] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1959.433785] Lustre: server umount lustre-MDT0000 complete [ 1960.755506] LustreError: 139371:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760453707 with bad export cookie 11526430513309691260 [ 1960.758682] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1960.759925] LustreError: 139371:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1960.897053] Lustre: server umount lustre-MDT0001 complete [ 1963.945568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1964.089918] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1964.110608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1965.523750] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 1968.502337] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1968.639896] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1970.172971] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 1972.891538] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1975.052052] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1977.434941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1977.549343] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1978.986123] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 1981.606658] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1983.284935] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1983.367073] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1984.925784] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 1985.016847] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1988.366666] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1988.390478] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1988.394288] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 1988.457985] Lustre: lustre-OST0001: new disk, initializing [ 1988.460321] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 1989.916573] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 1993.210669] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 1993.704622] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1993.708451] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1994.302447] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 1995.661862] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 1995.734340] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1997.647473] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9061526d1800.ost_server_uuid 40 [ 1998.150908] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9061526d1800.ost_server_uuid in FULL state after 0 sec [ 2003.937591] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2003.939267] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2003.941241] LustreError: Skipped 1 previous similar message [ 2003.944017] Lustre: Skipped 1 previous similar message [ 2003.944440] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2003.950815] Lustre: Skipped 3 previous similar messages [ 2006.061139] Lustre: server umount lustre-OST0001 complete [ 2009.059953] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2009.060149] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2009.062329] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2009.064150] Lustre: Skipped 2 previous similar messages [ 2009.067165] LustreError: Skipped 1 previous similar message [ 2013.533066] Lustre: server umount lustre-OST0000 complete [ 2019.297974] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2019.298693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2019.303675] Lustre: Skipped 1 previous similar message [ 2019.306049] Lustre: Skipped 1 previous similar message [ 2022.330103] LustreError: 147269:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760453769 with bad export cookie 11526430513309692100 [ 2022.333209] LustreError: 147269:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2022.333326] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2022.476423] Lustre: server umount lustre-MDT0001 complete [ 2022.478497] Lustre: Skipped 1 previous similar message [ 2030.120896] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 2031.219253] Key type lgssc unregistered [ 2031.343469] LNet: 150388:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2031.348652] LNet: Removed LNI 192.168.203.156@tcp [ 2031.672538] Key type .llcrypt unregistered [ 2031.673717] Key type ._llcrypt unregistered [ 2042.400972] alg: No test for adler32 (adler32-zlib) [ 2043.152362] Key type ._llcrypt registered [ 2043.153684] Key type .llcrypt registered [ 2043.190318] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_hostid [ 2049.148064] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 2049.653660] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2049.832047] Lustre: Lustre: Build Version: 2.15.7_5_g4a1d30d [ 2049.919803] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 2049.921847] LNet: Accept secure, port 988 [ 2051.512101] Key type lgssc registered [ 2051.905947] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2054.891891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2057.186851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2059.144793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2061.231259] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2063.559062] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2063.582072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2064.681788] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2064.692797] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2064.730853] Lustre: lustre-MDT0000: new disk, initializing [ 2064.756444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2064.763372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2066.144353] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 2070.171879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2070.189730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2070.219346] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2070.234398] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 2070.236130] Lustre: Skipped 1 previous similar message [ 2070.267111] Lustre: lustre-MDT0001: new disk, initializing [ 2070.294277] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2070.306962] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2070.310109] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2071.671828] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 2075.358359] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2077.293763] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2079.676993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2079.702627] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2079.705123] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2079.785650] Lustre: lustre-OST0000: new disk, initializing [ 2079.787509] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2079.811733] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2081.286552] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 2082.934124] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2082.937593] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2084.993421] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2086.419272] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2086.504590] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2087.946863] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2088.033402] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2093.538985] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2093.539526] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2093.541989] LustreError: Skipped 1 previous similar message [ 2093.546152] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2093.548682] Lustre: Skipped 1 previous similar message [ 2094.809699] Lustre: server umount lustre-OST0000 complete [ 2096.096955] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2096.099655] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2096.104310] Lustre: Skipped 1 previous similar message [ 2096.106706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2098.658291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2098.658486] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2102.153358] Lustre: server umount lustre-MDT0000 complete [ 2103.436721] LustreError: 155326:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760453850 with bad export cookie 16768029016507038829 [ 2103.440231] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2103.441792] LustreError: 155326:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2103.563689] Lustre: server umount lustre-MDT0001 complete [ 2106.319056] Lustre: DEBUG MARKER: == conf-sanity test 47: server restart does not make client loss lru_resize settings ========================================================== 10:57:33 (1760453853) [ 2111.362521] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_hostid [ 2114.152435] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 2117.210794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2119.256703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2121.184379] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2123.154662] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2125.436065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2125.460293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2125.541377] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2125.551397] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2125.587585] Lustre: lustre-MDT0000: new disk, initializing [ 2125.609646] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2125.617156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2127.060565] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 2131.079548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2131.105525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2131.137212] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2131.156078] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 2131.158479] Lustre: Skipped 1 previous similar message [ 2131.200248] Lustre: lustre-MDT0001: new disk, initializing [ 2131.229250] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2131.240867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2131.243856] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2132.606519] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 2136.194288] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2137.953310] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2140.077971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2140.101752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2140.104945] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2140.195313] Lustre: lustre-OST0000: new disk, initializing [ 2140.197962] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2140.221827] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2141.175847] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2141.179790] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2141.694791] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 2145.395442] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2147.376519] Lustre: Failing over lustre-OST0000 [ 2147.446127] Lustre: server umount lustre-OST0000 complete [ 2158.049194] Lustre: 152704:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760453898/real 1760453898] req@00000000799735f2 x1845969600670528/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1760453905 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 2158.057683] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2159.072178] Lustre: 152703:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760453898/real 1760453898] req@0000000079bd0e7e x1845969600670592/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1760453905 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 2159.079830] Lustre: 152703:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2160.824956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2160.955645] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2160.961924] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2162.375626] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 2162.976081] Lustre: 152701:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760453903/real 1760453903] req@0000000042c1c4ec x1845969600671488/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1760453910 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2163.636458] Lustre: Failing over lustre-MDT0000 [ 2163.814251] Lustre: server umount lustre-MDT0000 complete [ 2164.002063] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2164.003226] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2164.006283] Lustre: Skipped 3 previous similar messages [ 2164.011823] LustreError: Skipped 1 previous similar message [ 2165.025350] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2165.027087] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2165.029310] LustreError: Skipped 4 previous similar messages [ 2165.664116] Lustre: 152704:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760453905/real 1760453905] req@000000006a15020f x1845969600672064/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1760453912 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2170.336862] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2170.340410] LustreError: Skipped 3 previous similar messages [ 2172.384177] Lustre: 152702:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760453912/real 1760453912] req@00000000753b09bf x1845969600675200/t0(0) o400->MGC192.168.203.156@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1760453919 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2172.392312] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2172.397157] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2172.402387] LustreError: Skipped 3 previous similar messages [ 2176.543591] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2176.549549] LustreError: Skipped 1 previous similar message [ 2176.955357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2178.531720] Lustre: Evicted from MGS (at 192.168.203.156@tcp) after server handle changed from 0xe8b4018cb8583fb5 to 0xe8b4018cb8584454 [ 2178.537598] LustreError: 165767:0:(ldlm_resource.c:1127:ldlm_resource_complain()) MGC192.168.203.156@tcp: namespace resource [0x65727473756c:0x5:0x0].0x0 (00000000b88abe34) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 2178.542375] Lustre: MGC192.168.203.156@tcp: Connection restored to (at 0@lo) [ 2178.606714] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2178.653671] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2180.053232] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 2181.663627] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2183.657162] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 2183.662942] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 2183.667546] Lustre: Skipped 1 previous similar message [ 2183.685083] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 2188.771227] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2188.771380] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2188.772253] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2188.772258] Lustre: Skipped 1 previous similar message [ 2188.775237] Lustre: Skipped 1 previous similar message [ 2190.554501] Lustre: server umount lustre-OST0000 complete [ 2193.890383] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2193.890535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2193.892743] LustreError: Skipped 1 previous similar message [ 2193.895917] Lustre: Skipped 2 previous similar messages [ 2204.128938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2204.131297] Lustre: Skipped 3 previous similar messages [ 2206.176107] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2207.420926] LustreError: 161443:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760453954 with bad export cookie 16768029016507040852 [ 2207.423542] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2207.424817] LustreError: 161443:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2207.542280] Lustre: server umount lustre-MDT0001 complete [ 2207.543340] Lustre: Skipped 1 previous similar message [ 2212.082882] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 2213.040888] Key type lgssc unregistered [ 2213.150518] LNet: 167295:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2213.153707] LNet: Removed LNI 192.168.203.156@tcp [ 2213.458102] Key type .llcrypt unregistered [ 2213.459298] Key type ._llcrypt unregistered [ 2221.832614] alg: No test for adler32 (adler32-zlib) [ 2222.583376] Key type ._llcrypt registered [ 2222.585052] Key type .llcrypt registered [ 2222.620180] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 2223.066408] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2223.155601] Lustre: Lustre: Build Version: 2.15.7_5_g4a1d30d [ 2223.225816] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 2223.228272] LNet: Accept secure, port 988 [ 2224.808131] Key type lgssc registered [ 2225.103797] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2227.516037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2228.644250] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2228.656798] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2229.953391] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 2232.567440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2232.663794] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2234.225043] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 2236.690591] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2238.615014] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2240.908155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2241.022802] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2242.480538] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 2243.111124] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:65 [ 2245.060600] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2335.192978] Lustre: Failing over lustre-MDT0000 [ 2335.201107] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2335.203664] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2335.209346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2335.301843] Lustre: server umount lustre-MDT0000 complete [ 2338.343381] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2339.299316] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2339.303039] Lustre: Skipped 1 previous similar message [ 2343.455577] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2343.460345] LustreError: Skipped 3 previous similar messages [ 2345.514568] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2345.518690] LustreError: Skipped 3 previous similar messages [ 2346.337193] Lustre: 168070:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760454086/real 1760454086] req@000000002a653d05 x1845969782358016/t0(0) o400->MGC192.168.203.156@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1760454093 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2346.345866] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2348.445170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2350.623768] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2350.627878] LustreError: Skipped 3 previous similar messages [ 2352.610322] Lustre: Evicted from MGS (at 192.168.203.156@tcp) after server handle changed from 0x6e9b67e60f5adea3 to 0x6e9b67e60f5d4abd [ 2352.614946] LustreError: 171981:0:(ldlm_resource.c:1127:ldlm_resource_complain()) MGC192.168.203.156@tcp: namespace resource [0x65727473756c:0x5:0x0].0x0 (00000000a2f3c1a3) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 2352.619556] Lustre: MGC192.168.203.156@tcp: Connection restored to 192.168.203.156@tcp (at 0@lo) [ 2352.691677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2352.721120] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2353.982345] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 2356.767773] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2357.734767] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 2357.736894] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2357.757381] Lustre: lustre-OST0000: deleting orphan objects from 0x0:68 to 0x0:97 [ 2359.532153] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2360.040900] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2373.090038] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2373.090172] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2373.090775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2373.094238] Lustre: Skipped 1 previous similar message [ 2378.209908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2378.212673] Lustre: Skipped 4 previous similar messages [ 2378.820130] Lustre: server umount lustre-MDT0000 complete [ 2379.963846] LustreError: 168737:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760454127 with bad export cookie 7970078203360135869 [ 2379.965597] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2379.969122] LustreError: 168737:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2383.329105] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2383.329460] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2383.334394] LustreError: Skipped 7 previous similar messages [ 2383.341377] Lustre: Skipped 1 previous similar message [ 2383.342751] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2386.157333] Lustre: server umount lustre-MDT0001 complete [ 2393.564258] Lustre: server umount lustre-OST0000 complete [ 2396.440122] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_hostid [ 2399.319051] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 2402.546956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2404.566819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2406.430688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2408.372070] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2410.564114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2410.586183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2410.659635] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2410.669372] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2410.701381] Lustre: lustre-MDT0000: new disk, initializing [ 2410.721911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2410.728199] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2412.031237] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 2415.932573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2415.953726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2415.982498] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2415.998136] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 2415.999767] Lustre: Skipped 1 previous similar message [ 2416.033184] Lustre: lustre-MDT0001: new disk, initializing [ 2416.069183] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2416.071944] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2417.296420] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 2420.875645] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2422.588565] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2424.674854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2424.696621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2424.700161] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2424.783441] Lustre: lustre-OST0000: new disk, initializing [ 2424.784827] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2424.805253] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2424.807389] Lustre: Skipped 1 previous similar message [ 2426.064114] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 2429.651836] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2429.949277] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2429.952541] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2431.117573] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2431.190482] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2432.534579] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2432.611514] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2435.043546] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2435.043656] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2435.046146] LustreError: Skipped 1 previous similar message [ 2435.046840] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2435.050085] Lustre: Skipped 1 previous similar message [ 2435.055439] Lustre: Skipped 2 previous similar messages [ 2439.324083] Lustre: server umount lustre-OST0000 complete [ 2441.696843] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2445.281325] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2445.281640] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2445.282889] Lustre: Skipped 1 previous similar message [ 2445.288154] Lustre: Skipped 1 previous similar message [ 2447.837842] LustreError: 176470:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760454195 with bad export cookie 7970078203360136590 [ 2447.840381] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2447.841461] LustreError: 176470:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2447.960415] Lustre: server umount lustre-MDT0001 complete [ 2447.961408] Lustre: Skipped 1 previous similar message [ 2450.484775] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 11:03:17 (1760454197) [ 2455.450653] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_hostid [ 2457.992761] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 2460.792612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2462.789913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2464.689204] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2466.724894] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2468.830389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2468.852624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2468.932862] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2468.941968] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2468.981297] Lustre: lustre-MDT0000: new disk, initializing [ 2469.004627] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2469.009946] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2470.394608] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 2474.424811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2474.448404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2474.485972] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2474.505511] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 2474.508432] Lustre: Skipped 1 previous similar message [ 2474.546151] Lustre: lustre-MDT0001: new disk, initializing [ 2474.590715] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2474.595605] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2476.014983] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 2479.719559] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2481.468719] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2483.592787] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2483.614043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2483.616588] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2483.698735] Lustre: lustre-OST0000: new disk, initializing [ 2483.700574] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2485.100247] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 2488.864214] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2491.891396] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2491.894675] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2496.993395] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2496.994259] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2496.994867] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2496.994871] Lustre: Skipped 1 previous similar message [ 2496.996085] LustreError: Skipped 1 previous similar message [ 2496.999310] Lustre: Skipped 1 previous similar message [ 2498.258648] Lustre: server umount lustre-OST0000 complete [ 2502.112856] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2502.118568] LustreError: Skipped 2 previous similar messages [ 2502.718854] LustreError: 183454:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760454249 with bad export cookie 7970078203360137430 [ 2502.721871] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2502.722666] LustreError: 183454:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2505.363289] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 11:04:12 (1760454252) [ 2510.460150] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_hostid [ 2513.204095] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 2516.229563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2518.300328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2520.098583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2521.948635] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2524.069043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2524.092211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2524.162425] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2524.170788] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2524.203742] Lustre: lustre-MDT0000: new disk, initializing [ 2524.224541] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2524.226770] Lustre: Skipped 2 previous similar messages [ 2524.232361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2525.597210] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 2529.427686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2529.449994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2529.566343] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2530.894650] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 2534.511282] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2536.204683] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2538.274049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2538.300864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2538.304098] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2539.722085] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 2540.917763] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2543.440176] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2546.146308] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2546.146707] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2546.149325] LustreError: Skipped 2 previous similar messages [ 2546.154742] Lustre: Skipped 2 previous similar messages [ 2546.155364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2546.158853] Lustre: Skipped 3 previous similar messages [ 2551.314935] Lustre: server umount lustre-OST0000 complete [ 2551.316606] Lustre: Skipped 2 previous similar messages [ 2559.815152] LustreError: 189907:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760454307 with bad export cookie 7970078203360138494 [ 2559.817866] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2559.818220] LustreError: 189907:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2564.624874] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 2565.607085] Key type lgssc unregistered [ 2565.712374] LNet: 193008:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2565.714585] LNet: Removed LNI 192.168.203.156@tcp [ 2565.972172] Key type .llcrypt unregistered [ 2565.972961] Key type ._llcrypt unregistered [ 2574.388041] alg: No test for adler32 (adler32-zlib) [ 2575.139310] Key type ._llcrypt registered [ 2575.140126] Key type .llcrypt registered [ 2575.170011] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 2575.538692] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2575.665888] Lustre: Lustre: Build Version: 2.15.7_5_g4a1d30d [ 2575.730473] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 2575.732196] LNet: Accept secure, port 988 [ 2577.304053] Key type lgssc registered [ 2577.606498] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2579.946697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2581.076152] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2581.087793] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2582.540472] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 2585.222645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2585.335627] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2586.721589] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 2589.085190] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2590.910417] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2593.021678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2593.103276] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2594.271800] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 2596.198937] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 2596.751813] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2615.778131] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2615.778605] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2615.782906] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2615.787538] Lustre: Skipped 1 previous similar message [ 2619.351081] Lustre: server umount lustre-OST0000 complete [ 2620.897439] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2620.898157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2620.900848] Lustre: Skipped 2 previous similar messages [ 2620.905061] Lustre: Skipped 1 previous similar message [ 2626.017266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2626.019257] Lustre: Skipped 1 previous similar message [ 2626.827088] Lustre: server umount lustre-MDT0000 complete [ 2628.121153] LustreError: 194449:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760454375 with bad export cookie 4980795208146993674 [ 2628.123856] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2628.126140] LustreError: 194449:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2628.246237] Lustre: server umount lustre-MDT0001 complete [ 2633.033449] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 2634.070265] Key type lgssc unregistered [ 2634.189380] LNet: 198142:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2634.194552] LNet: Removed LNI 192.168.203.156@tcp [ 2634.492199] Key type .llcrypt unregistered [ 2634.493184] Key type ._llcrypt unregistered [ 2643.525531] alg: No test for adler32 (adler32-zlib) [ 2644.277315] Key type ._llcrypt registered [ 2644.278455] Key type .llcrypt registered [ 2644.316992] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 2644.647741] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2644.869419] Lustre: Lustre: Build Version: 2.15.7_5_g4a1d30d [ 2644.947435] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 2644.949930] LNet: Accept secure, port 988 [ 2646.528148] Key type lgssc registered [ 2646.866063] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2649.573370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2650.726520] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2650.741847] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2652.135817] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 2654.829924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2654.936497] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2656.312372] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 2658.627270] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2660.427044] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2662.443407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2662.538649] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2663.821587] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 2666.112400] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2666.661393] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 2676.194777] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2676.194829] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2676.202376] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2676.204553] Lustre: Skipped 1 previous similar message [ 2677.799261] Lustre: lustre-OST0000: Not available for connect from 192.168.203.56@tcp (stopping) [ 2680.726403] Lustre: server umount lustre-OST0000 complete [ 2683.106393] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff906152022800.ost_server_uuid 40 [ 2683.644224] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 2696.160618] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2696.162713] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2696.165697] Lustre: Skipped 1 previous similar message [ 2696.167316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2697.697865] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2699.718724] Lustre: server umount lustre-MDT0000 complete [ 2700.892946] LustreError: 199583:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760454448 with bad export cookie 15820682823860274558 [ 2700.895134] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2700.897074] LustreError: 199583:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2701.013971] Lustre: server umount lustre-MDT0001 complete [ 2703.614187] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 11:07:30 (1760454450) [ 2705.712578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2705.845926] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2705.866971] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2707.221240] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 2710.146827] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2710.258168] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2711.674274] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 2713.973461] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2715.777707] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2717.812925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2719.269977] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 2720.997712] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 2721.546860] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2723.662907] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2723.688994] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2723.692560] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 2723.744463] Lustre: lustre-OST0001: new disk, initializing [ 2723.746821] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 2723.772352] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2723.774189] Lustre: Skipped 1 previous similar message [ 2725.131784] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 2728.662369] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2728.957559] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2728.960361] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2729.952772] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2729.954858] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2729.958117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2729.959683] Lustre: Skipped 2 previous similar messages [ 2734.051639] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2734.054352] Lustre: Skipped 2 previous similar messages [ 2735.532271] Lustre: server umount lustre-OST0000 complete [ 2737.395406] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2737.476823] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 2738.851867] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2738.926495] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 2739.173788] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2739.178211] LustreError: Skipped 1 previous similar message [ 2744.291955] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2744.295211] LustreError: Skipped 2 previous similar messages [ 2749.410022] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2749.410317] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2749.412533] LustreError: Skipped 1 previous similar message [ 2749.412554] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2749.416469] LustreError: Skipped 2 previous similar messages [ 2749.417808] Lustre: Skipped 1 previous similar message [ 2749.424698] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2749.427034] Lustre: Skipped 2 previous similar messages [ 2754.983048] Lustre: server umount lustre-OST0001 complete [ 2756.576822] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2756.578950] LustreError: Skipped 1 previous similar message [ 2759.650208] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2759.653962] Lustre: Skipped 2 previous similar messages [ 2763.681129] LustreError: 203548:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760454510 with bad export cookie 15820682823860275608 [ 2763.684610] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2763.686830] LustreError: 203548:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2763.821767] Lustre: server umount lustre-MDT0001 complete [ 2763.823580] Lustre: Skipped 1 previous similar message [ 2765.612278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2766.832074] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2768.104896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2769.414275] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2771.468191] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 11:08:38 (1760454518) [ 2773.560233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2773.671191] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2773.684155] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2773.755211] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2775.230476] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 2777.975674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2778.008930] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2778.022792] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2779.563286] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 2781.978539] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2783.779981] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2786.035642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2786.113785] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2787.466026] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 2789.873231] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2792.189615] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2792.216075] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2793.732145] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 2795.369853] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 2796.245130] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2800.612925] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2800.613478] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2800.614621] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2800.614625] Lustre: Skipped 5 previous similar messages [ 2800.616048] LustreError: Skipped 1 previous similar message [ 2800.619184] Lustre: Skipped 1 previous similar message [ 2803.243570] Lustre: server umount lustre-OST0000 complete [ 2805.733140] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2805.739584] LustreError: Skipped 3 previous similar messages [ 2824.160918] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2824.163996] LustreError: Skipped 2 previous similar messages [ 2828.523829] LustreError: 209955:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760454575 with bad export cookie 15820682823860276973 [ 2828.527192] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2828.527959] LustreError: 209955:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2830.472935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2831.682130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2832.892359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2834.122499] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2836.247316] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 11:09:43 (1760454583) [ 2841.102039] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_hostid [ 2843.672514] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 2846.772516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2848.738139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2850.589386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2852.463572] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2854.529062] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2854.550123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2854.627537] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2854.636648] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2854.668124] Lustre: lustre-MDT0000: new disk, initializing [ 2854.687176] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2854.688759] Lustre: Skipped 3 previous similar messages [ 2854.694144] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2855.995303] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 2860.025400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2860.048517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2860.078437] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2860.093641] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 2860.095550] Lustre: Skipped 1 previous similar message [ 2860.128657] Lustre: lustre-MDT0001: new disk, initializing [ 2860.170844] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2860.174031] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2861.479453] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 2865.147922] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2866.984166] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2869.182866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2869.204618] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2869.208432] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2869.291968] Lustre: lustre-OST0000: new disk, initializing [ 2869.293497] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2870.687709] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 2873.333357] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2873.335764] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2874.291728] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2875.613098] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2875.673421] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2876.967806] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2877.036443] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2878.434548] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2878.435483] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2878.437960] LustreError: Skipped 1 previous similar message [ 2878.442268] Lustre: Skipped 4 previous similar messages [ 2878.444197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2878.446383] Lustre: Skipped 8 previous similar messages [ 2879.713508] Lustre: server umount lustre-OST0000 complete [ 2879.714844] Lustre: Skipped 3 previous similar messages [ 2895.328123] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2896.642138] LustreError: 218151:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760454643 with bad export cookie 15820682823860278373 [ 2896.644984] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2896.646826] LustreError: 218151:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2899.521902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2899.665584] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2899.669854] LustreError: Skipped 6 previous similar messages [ 2901.001357] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 2903.799701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2905.265085] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 2907.667572] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2909.527225] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2911.730429] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2913.345240] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 2915.740995] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2917.171762] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2917.237081] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2918.562609] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2919.659810] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 1 sec [ 2928.419361] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2928.491099] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 2930.038358] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2953.623203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2953.726653] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2953.728740] Lustre: Skipped 5 previous similar messages [ 2955.104171] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 2955.811761] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2955.816888] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to (at 0@lo) [ 2955.819748] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:65 [ 2957.630915] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2962.324519] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 4 sec [ 2965.985577] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2965.987502] LustreError: Skipped 3 previous similar messages [ 2965.988621] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2965.991228] Lustre: Skipped 4 previous similar messages [ 2965.992704] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2965.994087] Lustre: Skipped 12 previous similar messages [ 2969.449150] Lustre: server umount lustre-OST0000 complete [ 2969.451228] Lustre: Skipped 3 previous similar messages [ 2977.249030] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2977.255018] LustreError: Skipped 1 previous similar message [ 2978.090827] LustreError: 222496:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760454725 with bad export cookie 15820682823860279213 [ 2978.093445] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2978.095910] LustreError: 222496:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2981.109352] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 11:12:08 (1760454728) [ 2983.430743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2985.204820] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 2988.011153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2989.613085] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 2992.184740] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2994.183941] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2996.510977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2997.994328] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 3000.543878] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3002.077981] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3002.165019] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3003.630263] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3004.726942] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 1 sec [ 3006.969175] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3007.003174] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3007.007305] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 3007.063828] Lustre: lustre-OST0001: new disk, initializing [ 3007.065552] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 3008.498933] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 3009.210855] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3009.215348] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3012.329922] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3013.813637] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 3013.906188] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3015.364550] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 3015.456494] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3024.190633] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 3024.265880] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3025.752783] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 3025.845570] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3041.313290] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 192.168.203.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3041.318959] LustreError: Skipped 12 previous similar messages [ 3049.377582] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3050.887873] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 3053.411878] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3053.928270] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3087.430841] LustreError: 228705:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760454834 with bad export cookie 15820682823860280375 [ 3087.433608] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3087.435326] LustreError: 228705:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3095.564604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3096.864181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3098.116915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3099.416993] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3101.525911] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 11:14:08 (1760454848) [ 3103.675586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3103.765615] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3103.774278] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3103.839172] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3103.840847] Lustre: Skipped 5 previous similar messages [ 3105.106284] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 3107.840749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3107.870684] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3109.325906] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 3111.767245] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3113.759358] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3115.938666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3116.015233] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3117.459228] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 3119.963991] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3128.828341] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3128.858679] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3130.316244] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 3132.959182] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3136.635207] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec [ 3138.011245] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 3138.078614] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3139.448916] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 3139.529208] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3141.447649] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff906159756800.ost_server_uuid 40 [ 3141.943153] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff906159756800.ost_server_uuid in FULL state after 0 sec [ 3142.282961] Lustre: Permanently deactivating lustre-OST0001 [ 3142.288622] Lustre: Setting parameter lustre-OST0001-osc.osc.active in log lustre-client [ 3142.291290] Lustre: Skipped 1 previous similar message [ 3143.107805] Lustre: Permanently reactivating lustre-OST0001 [ 3145.186101] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3145.188350] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3145.189360] LustreError: Skipped 4 previous similar messages [ 3145.195906] Lustre: Skipped 8 previous similar messages [ 3145.196639] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3145.200096] Lustre: Skipped 20 previous similar messages [ 3150.121399] Lustre: server umount lustre-OST0001 complete [ 3150.123340] Lustre: Skipped 7 previous similar messages [ 3173.344118] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3174.605230] LustreError: 237620:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760454921 with bad export cookie 15820682823860281621 [ 3174.608094] LustreError: 237620:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3174.608404] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3176.625982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3177.897477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3179.104683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3180.352371] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3182.491377] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 11:15:29 (1760454929) [ 3182.839765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3184.999022] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3185.097311] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3185.107332] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3185.110163] Lustre: Skipped 5 previous similar messages [ 3186.581802] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 3189.330039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3189.362517] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3190.876880] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 3193.416319] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3195.358800] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3197.551681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3197.573883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3197.644207] Lustre: Permanently deactivating lustre-OST0000 [ 3197.697957] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3199.143984] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 3201.643559] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3203.846313] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3203.895254] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3203.899908] LustreError: Skipped 13 previous similar messages [ 3205.446509] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 3208.037193] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3209.192306] Lustre: lustre-OST0001: deleting orphan objects from 0x0:35 to 0x0:65 [ 3213.779354] Lustre: Permanently reactivating lustre-OST0000 [ 3215.042709] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3215.049276] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 3238.490965] LustreError: 246449:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760454985 with bad export cookie 15820682823860283203 [ 3238.493622] LustreError: 246449:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3238.493857] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3243.079973] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 3243.968302] Key type lgssc unregistered [ 3244.075312] LNet: 249408:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3244.077492] LNet: Removed LNI 192.168.203.156@tcp [ 3244.371771] Key type .llcrypt unregistered [ 3244.372799] Key type ._llcrypt unregistered [ 3252.765200] alg: No test for adler32 (adler32-zlib) [ 3253.516358] Key type ._llcrypt registered [ 3253.517236] Key type .llcrypt registered [ 3253.551072] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 3253.826302] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3254.074791] Lustre: Lustre: Build Version: 2.15.7_5_g4a1d30d [ 3254.145749] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 3254.147434] LNet: Accept secure, port 988 [ 3255.720145] Key type lgssc registered [ 3256.067380] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3257.284982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3259.414762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3260.552611] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3260.570244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3261.938239] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 3264.652333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3264.675157] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3264.708234] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3264.717096] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3264.718960] Lustre: Permanently deactivating lustre-MDT0001 [ 3264.812735] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3266.191396] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 3267.181727] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3268.766768] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3270.715779] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3272.891761] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3272.993392] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3274.346711] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 3276.068372] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3276.070190] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:161 [ 3276.851970] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3279.117375] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3279.181441] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3280.546108] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 3281.255688] Lustre: lustre-OST0001: deleting orphan objects from 0x0:35 to 0x0:97 [ 3281.256762] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:163 to 0x280000400:193 [ 3281.257118] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:67 to 0x2c0000400:97 [ 3282.941876] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3284.464779] Lustre: Permanently reactivating lustre-MDT0001 [ 3284.471829] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active in log lustre-client [ 3284.473494] Lustre: Skipped 3 previous similar messages [ 3289.615276] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3289.622059] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 3289.624778] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 3289.630280] LustreError: 254514:0:(llog_osd.c:2141:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 3289.630641] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to (at 0@lo) [ 3289.633127] LustreError: 254514:0:(lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 3289.639246] LustreError: 254514:0:(obd_config.c:1999:class_config_llog_handler()) MGC192.168.203.156@tcp: cfg command failed: rc = -108 [ 3289.644484] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 3289.644484] [ 3289.648544] LustreError: 250909:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 3291.173789] Lustre: lustre-MDT0001: Client dab52a87-f908-41e2-a06d-15a172feec2d (at 192.168.203.56@tcp) reconnecting [ 3293.522951] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3294.094901] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3296.160100] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 40 [ 3296.249480] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 3297.668539] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 40 [ 3297.748732] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 3298.541783] Lustre: Permanently deactivating lustre-MDT0001 [ 3298.546945] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active in log lustre-client [ 3298.548403] Lustre: Skipped 2 previous similar messages [ 3305.470806] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3309.986520] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3309.989513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3309.995577] Lustre: Skipped 3 previous similar messages [ 3309.999679] Lustre: Skipped 2 previous similar messages [ 3313.444629] Lustre: server umount lustre-MDT0000 complete [ 3315.171387] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3315.179398] LustreError: Skipped 3 previous similar messages [ 3315.222510] LustreError: 250899:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760455062 with bad export cookie 9105027754623093996 [ 3315.227499] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3315.397987] Lustre: server umount lustre-MDT0001 complete [ 3323.308547] Lustre: server umount lustre-OST0000 complete [ 3327.079868] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 11:17:54 (1760455074) [ 3331.240650] Lustre: server umount lustre-OST0001 complete [ 3334.009864] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_hostid [ 3337.330531] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 3341.298650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3343.613206] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3346.393415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3349.272353] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3352.192930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3352.215421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3352.311681] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3352.313711] Lustre: Skipped 2 previous similar messages [ 3352.323305] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3352.370494] Lustre: lustre-MDT0000: new disk, initializing [ 3352.398533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3352.406087] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3354.014683] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 3359.292227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3359.324167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3359.362286] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3359.390536] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3359.394623] Lustre: Skipped 1 previous similar message [ 3359.447147] Lustre: lustre-MDT0001: new disk, initializing [ 3359.510407] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3359.517747] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3361.202977] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 3365.974789] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3368.129865] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3370.852610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3370.891123] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3370.893352] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3371.012044] Lustre: lustre-OST0000: new disk, initializing [ 3371.013928] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3371.054283] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3371.058972] Lustre: Skipped 1 previous similar message [ 3372.848124] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 3377.691133] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3380.727314] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3380.730215] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3384.087518] LustreError: 259395:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3388.303370] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3388.325302] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3388.328423] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 3388.380603] Lustre: lustre-OST0001: new disk, initializing [ 3388.382585] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 3388.408725] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3390.076637] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 3394.176131] LustreError: 259395:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 awake [ 3394.187858] LustreError: 259395:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3394.589134] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3395.248309] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3398.669417] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3398.673028] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3404.280166] LustreError: 259395:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 awake [ 3406.894028] Lustre: lustre-OST0001: Not available for connect from 192.168.203.56@tcp (stopping) [ 3408.868861] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3408.868979] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3408.871890] LustreError: Skipped 1 previous similar message [ 3408.881077] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3408.883824] Lustre: Skipped 1 previous similar message [ 3411.120651] Lustre: server umount lustre-OST0001 complete [ 3412.000216] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 192.168.203.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3419.107352] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3419.112115] LustreError: Skipped 3 previous similar messages [ 3424.227268] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3424.231833] Lustre: Skipped 1 previous similar message [ 3424.233315] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3424.236384] Lustre: Skipped 1 previous similar message [ 3429.274548] Lustre: server umount lustre-OST0000 complete [ 3431.392774] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3431.396628] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3431.402353] Lustre: Skipped 1 previous similar message [ 3431.404689] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3437.917313] LustreError: 259381:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760455185 with bad export cookie 9105027754623096747 [ 3437.920267] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3437.921161] LustreError: 259381:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 3452.384107] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3452.478316] Lustre: server umount lustre-MDT0001 complete [ 3452.479513] Lustre: Skipped 1 previous similar message [ 3456.860768] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 3457.923437] Key type lgssc unregistered [ 3458.038362] LNet: 264589:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3458.040830] LNet: Removed LNI 192.168.203.156@tcp [ 3458.359532] Key type .llcrypt unregistered [ 3458.360614] Key type ._llcrypt unregistered [ 3469.646214] alg: No test for adler32 (adler32-zlib) [ 3470.398299] Key type ._llcrypt registered [ 3470.399196] Key type .llcrypt registered [ 3470.438301] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_hostid [ 3476.529816] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 3476.928485] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3477.181094] Lustre: Lustre: Build Version: 2.15.7_5_g4a1d30d [ 3477.257474] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 3477.259397] LNet: Accept secure, port 988 [ 3478.840116] Key type lgssc registered [ 3479.198785] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3482.109483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3484.209426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3486.207950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3488.300435] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3490.721514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3490.754563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3491.867347] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3491.877703] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3491.917490] Lustre: lustre-MDT0000: new disk, initializing [ 3491.942953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3491.950445] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3493.341516] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 3497.329800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3497.353716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3497.390540] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3497.408556] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3497.410503] Lustre: Skipped 1 previous similar message [ 3497.450068] Lustre: lustre-MDT0001: new disk, initializing [ 3497.477945] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3497.490632] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3497.493462] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3498.915152] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 3502.734425] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3504.547389] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3506.781477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3506.808762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3506.811325] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3506.913450] Lustre: lustre-OST0000: new disk, initializing [ 3506.915185] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3506.943258] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3508.374372] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 3508.597623] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3508.600526] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3512.237152] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3513.728198] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3513.814589] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3515.352034] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3515.436278] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3518.947424] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3518.948359] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3518.949269] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3518.952177] Lustre: Skipped 1 previous similar message [ 3518.955624] LustreError: Skipped 1 previous similar message [ 3522.213816] Lustre: server umount lustre-OST0000 complete [ 3524.065503] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3524.065956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3524.070976] Lustre: Skipped 1 previous similar message [ 3524.074102] Lustre: Skipped 1 previous similar message [ 3529.184862] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3529.187291] Lustre: Skipped 1 previous similar message [ 3529.603390] Lustre: server umount lustre-MDT0000 complete [ 3530.827178] LustreError: 268671:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760455278 with bad export cookie 5119884280723372621 [ 3530.830186] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3530.831939] LustreError: 268671:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3530.965136] Lustre: server umount lustre-MDT0001 complete [ 3533.618309] Lustre: DEBUG MARKER: == conf-sanity test 52: check recovering objects from lost+found ========================================================== 11:21:20 (1760455280) [ 3535.737762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3535.883342] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3535.904223] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3537.273956] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 3540.069031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3540.187071] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3541.622548] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 3544.093691] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3545.994085] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3548.249973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3548.354339] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3549.717365] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 3552.212926] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3561.953167] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3561.955370] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3561.958780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3566.052077] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3566.052191] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3566.055874] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3567.713669] Lustre: server umount lustre-OST0000 complete [ 3569.172859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 3573.138969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3573.295433] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3575.318543] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 3579.071769] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3582.502528] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3582.508164] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.203.156@tcp (at 0@lo) [ 3582.510234] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:65 [ 3587.555369] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3587.557461] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3587.562842] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3587.569554] Lustre: Skipped 1 previous similar message [ 3588.848908] Lustre: server umount lustre-OST0000 complete [ 3591.649262] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3598.034680] LustreError: 273208:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760455345 with bad export cookie 5119884280723373461 [ 3598.040725] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3598.041101] LustreError: 273208:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3598.178700] Lustre: server umount lustre-MDT0001 complete [ 3598.180062] Lustre: Skipped 1 previous similar message [ 3603.318173] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 3604.322722] Key type lgssc unregistered [ 3604.436316] LNet: 278633:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3604.439144] LNet: Removed LNI 192.168.203.156@tcp [ 3604.721459] Key type .llcrypt unregistered [ 3604.723283] Key type ._llcrypt unregistered [ 3614.338593] alg: No test for adler32 (adler32-zlib) [ 3615.090321] Key type ._llcrypt registered [ 3615.091230] Key type .llcrypt registered [ 3615.128789] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 3615.562987] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3615.669447] Lustre: Lustre: Build Version: 2.15.7_5_g4a1d30d [ 3615.742158] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 3615.744426] LNet: Accept secure, port 988 [ 3617.320106] Key type lgssc registered [ 3617.938407] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3621.051845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3622.223741] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3622.240079] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3623.753851] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 3627.490092] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3627.561161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3627.703157] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3629.331390] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 3632.842491] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3635.052202] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3638.261148] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3638.396202] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3639.999363] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 3642.512622] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3652.713164] Lustre: lustre-OST0000: deleting orphan objects from 0x0:42 to 0x0:65 [ 3663.333408] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3663.334080] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3663.343711] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3663.354360] Lustre: Skipped 1 previous similar message [ 3664.219792] Lustre: server umount lustre-OST0000 complete [ 3668.449278] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3668.449813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3668.452112] Lustre: Skipped 2 previous similar messages [ 3668.456152] Lustre: Skipped 1 previous similar message [ 3671.565571] Lustre: server umount lustre-MDT0000 complete [ 3672.821627] LustreError: 280073:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760455420 with bad export cookie 6373812061159742545 [ 3672.823719] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3672.824743] LustreError: 280073:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3672.968189] Lustre: server umount lustre-MDT0001 complete [ 3677.315614] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 3678.315491] Key type lgssc unregistered [ 3678.433383] LNet: 284537:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3678.435593] LNet: Removed LNI 192.168.203.156@tcp [ 3678.713584] Key type .llcrypt unregistered [ 3678.715563] Key type ._llcrypt unregistered [ 3685.578982] alg: No test for adler32 (adler32-zlib) [ 3686.330456] Key type ._llcrypt registered [ 3686.332202] Key type .llcrypt registered [ 3686.364715] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 3686.676193] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3686.895180] Lustre: Lustre: Build Version: 2.15.7_5_g4a1d30d [ 3686.965206] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 3686.966601] LNet: Accept secure, port 988 [ 3688.536086] Key type lgssc registered [ 3688.886079] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3691.396445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3692.534024] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3692.550521] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3693.912144] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 3696.655636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3696.777941] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3698.180926] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 3700.694602] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3702.589306] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3704.763637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3704.884959] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3706.306910] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 3707.941304] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 3708.772244] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3718.115848] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3718.116053] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3718.121469] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3718.122853] Lustre: Skipped 1 previous similar message [ 3723.224895] Lustre: server umount lustre-OST0000 complete [ 3727.840702] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3727.842787] LustreError: Skipped 1 previous similar message [ 3727.844139] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3727.847887] Lustre: Skipped 1 previous similar message [ 3727.849633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3730.571851] Lustre: server umount lustre-MDT0000 complete [ 3731.811102] LustreError: 285828:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760455479 with bad export cookie 2194019320755891948 [ 3731.814020] LustreError: 285828:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3731.814383] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3731.950848] Lustre: server umount lustre-MDT0001 complete [ 3736.509366] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 3737.428352] Key type lgssc unregistered [ 3737.543340] LNet: 289624:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3737.546290] LNet: Removed LNI 192.168.203.156@tcp [ 3737.834916] Key type .llcrypt unregistered [ 3737.835881] Key type ._llcrypt unregistered [ 3745.462298] alg: No test for adler32 (adler32-zlib) [ 3746.214315] Key type ._llcrypt registered [ 3746.215155] Key type .llcrypt registered [ 3746.265495] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 3746.655377] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3746.895935] Lustre: Lustre: Build Version: 2.15.7_5_g4a1d30d [ 3746.981430] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 3746.983730] LNet: Accept secure, port 988 [ 3748.576110] Key type lgssc registered [ 3749.142683] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3752.332589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3753.481296] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3753.496353] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3754.971784] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 3758.562409] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3758.767991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3758.947102] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3760.870465] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 3764.308821] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3766.677106] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3769.940781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3770.111721] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3771.505929] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 3774.408415] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3783.468807] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 3788.770941] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3788.770984] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3788.771556] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3788.774204] Lustre: Skipped 1 previous similar message [ 3789.848804] Lustre: server umount lustre-OST0000 complete [ 3793.889615] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3793.890273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3793.892995] Lustre: Skipped 1 previous similar message [ 3793.894888] Lustre: Skipped 1 previous similar message [ 3797.266224] Lustre: server umount lustre-MDT0000 complete [ 3798.517960] LustreError: 292839:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760455545 with bad export cookie 5121563620133177133 [ 3798.520301] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3798.522129] LustreError: 292839:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3798.643669] Lustre: server umount lustre-MDT0001 complete [ 3803.417245] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 3804.304804] Key type lgssc unregistered [ 3804.411314] LNet: 294568:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3804.413739] LNet: Removed LNI 192.168.203.156@tcp [ 3804.683770] Key type .llcrypt unregistered [ 3804.684874] Key type ._llcrypt unregistered [ 3813.098812] alg: No test for adler32 (adler32-zlib) [ 3813.849323] Key type ._llcrypt registered [ 3813.850233] Key type .llcrypt registered [ 3813.894184] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 3814.309453] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3814.571721] Lustre: Lustre: Build Version: 2.15.7_5_g4a1d30d [ 3814.646888] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 3814.648256] LNet: Accept secure, port 988 [ 3816.224148] Key type lgssc registered [ 3816.627468] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3819.271794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3820.410738] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3820.426572] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3821.878041] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 3824.891977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3824.995667] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3826.325147] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 3828.835601] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3830.755786] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3832.979769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3833.086629] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3834.434108] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 3836.197578] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:161 [ 3836.827485] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3850.209743] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3850.211649] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3850.214810] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3850.721902] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3850.723552] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3850.729884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3855.133674] Lustre: server umount lustre-OST0000 complete [ 3860.961344] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3860.961497] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3860.961990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3860.969316] Lustre: Skipped 1 previous similar message [ 3862.476114] Lustre: server umount lustre-MDT0000 complete [ 3863.890159] LustreError: 296008:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760455611 with bad export cookie 11876641358131428354 [ 3863.892964] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3863.893604] LustreError: 296008:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3864.032609] Lustre: server umount lustre-MDT0001 complete [ 3868.881474] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 3869.780148] Key type lgssc unregistered [ 3869.901395] LNet: 300499:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3869.903822] LNet: Removed LNI 192.168.203.156@tcp [ 3870.198321] Key type .llcrypt unregistered [ 3870.199484] Key type ._llcrypt unregistered [ 3877.269474] alg: No test for adler32 (adler32-zlib) [ 3878.020333] Key type ._llcrypt registered [ 3878.021241] Key type .llcrypt registered [ 3878.051935] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 3878.439554] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3878.577387] Lustre: Lustre: Build Version: 2.15.7_5_g4a1d30d [ 3878.644922] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 3878.646526] LNet: Accept secure, port 988 [ 3880.216186] Key type lgssc registered [ 3880.744791] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3883.419359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3884.574800] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3884.587572] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3885.933125] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 3888.876293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3889.039868] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3890.559471] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 3893.195509] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3895.270257] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3897.570099] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3897.683237] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3899.137189] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 3900.773924] Lustre: lustre-OST0000: deleting orphan objects from 0x0:163 to 0x0:193 [ 3901.928724] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3911.136826] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3911.139102] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3911.143096] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3914.210671] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3914.211143] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3914.214585] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3916.333337] Lustre: server umount lustre-OST0000 complete [ 3919.329609] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3919.330085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3919.333438] Lustre: Skipped 1 previous similar message [ 3919.334913] Lustre: Skipped 1 previous similar message [ 3924.448927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3924.451587] Lustre: Skipped 2 previous similar messages [ 3929.568897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3929.570482] Lustre: Skipped 1 previous similar message [ 3932.128140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3932.166228] Lustre: server umount lustre-MDT0000 complete [ 3933.400420] LustreError: 301746:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760455680 with bad export cookie 3305974294784771822 [ 3933.403466] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3933.405174] LustreError: 301746:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3933.540783] Lustre: server umount lustre-MDT0001 complete [ 3938.170676] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 3939.136908] Key type lgssc unregistered [ 3939.266446] LNet: 305602:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3939.268788] LNet: Removed LNI 192.168.203.156@tcp [ 3939.593021] Key type .llcrypt unregistered [ 3939.593958] Key type ._llcrypt unregistered [ 3946.270163] alg: No test for adler32 (adler32-zlib) [ 3947.021478] Key type ._llcrypt registered [ 3947.023333] Key type .llcrypt registered [ 3947.069966] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 3947.435612] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3947.676928] Lustre: Lustre: Build Version: 2.15.7_5_g4a1d30d [ 3947.750050] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 3947.752318] LNet: Accept secure, port 988 [ 3949.328095] Key type lgssc registered [ 3949.667781] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3952.247276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3953.397312] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3953.412369] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3954.863043] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 3957.683548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3957.803105] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3959.163409] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 3961.499220] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3963.337338] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3965.522782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3965.650142] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3967.114862] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 3969.548278] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3969.768083] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:225 [ 3979.234847] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3979.235196] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3979.240790] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3979.243386] Lustre: Skipped 1 previous similar message [ 3984.354725] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3984.357459] Lustre: Skipped 1 previous similar message [ 3984.663656] Lustre: server umount lustre-OST0000 complete [ 3988.448987] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3988.452913] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3988.459370] Lustre: Skipped 1 previous similar message [ 3988.462115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3989.473588] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3994.593025] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3994.594674] Lustre: Skipped 3 previous similar messages [ 3999.712952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3999.714600] Lustre: Skipped 1 previous similar message [ 4000.224090] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 4000.264063] Lustre: server umount lustre-MDT0000 complete [ 4001.487945] LustreError: 306863:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760455748 with bad export cookie 2901850720575159151 [ 4001.490563] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4001.491509] LustreError: 306863:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4001.625219] Lustre: server umount lustre-MDT0001 complete [ 4006.095082] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 4007.111577] Key type lgssc unregistered [ 4007.230382] LNet: 310544:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4007.232739] LNet: Removed LNI 192.168.203.156@tcp [ 4007.542725] Key type .llcrypt unregistered [ 4007.543922] Key type ._llcrypt unregistered [ 4012.126380] alg: No test for adler32 (adler32-zlib) [ 4012.877373] Key type ._llcrypt registered [ 4012.878327] Key type .llcrypt registered [ 4012.916654] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 4018.428976] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_hostid [ 4023.602352] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 4024.084246] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4024.251791] Lustre: Lustre: Build Version: 2.15.7_5_g4a1d30d [ 4024.335029] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 4024.337271] LNet: Accept secure, port 988 [ 4025.920143] Key type lgssc registered [ 4026.291663] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4029.345096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4031.845977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4033.781154] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4036.067684] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4038.837929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4038.863905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4039.953515] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4039.966886] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4040.006847] Lustre: lustre-MDT0000: new disk, initializing [ 4040.035204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4040.043749] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4041.542863] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 4046.033629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4046.055346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4046.089637] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4046.104947] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 4046.106617] Lustre: Skipped 1 previous similar message [ 4046.140593] Lustre: lustre-MDT0001: new disk, initializing [ 4046.168763] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4046.180518] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4046.183867] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4047.576804] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 4051.432764] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4053.362971] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4055.723841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4055.757647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4055.761419] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4055.877452] Lustre: lustre-OST0000: new disk, initializing [ 4055.879986] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 4055.910464] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4056.635609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4056.638641] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4057.452176] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 4061.448974] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4063.038289] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 4063.106000] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4064.723573] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 4064.823367] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4066.785820] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4066.786976] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4066.793601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4066.795268] Lustre: Skipped 1 previous similar message [ 4067.638497] Lustre: server umount lustre-OST0000 complete [ 4071.905429] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4071.905730] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4071.911731] Lustre: Skipped 1 previous similar message [ 4071.913210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4071.914557] Lustre: Skipped 1 previous similar message [ 4077.024904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4077.026627] Lustre: Skipped 1 previous similar message [ 4082.145126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4082.146863] Lustre: Skipped 1 previous similar message [ 4083.168174] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 4083.213381] Lustre: server umount lustre-MDT0000 complete [ 4084.575172] LustreError: 315139:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760455831 with bad export cookie 15858582668421966377 [ 4084.577950] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4084.578008] LustreError: 315139:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4084.699665] Lustre: server umount lustre-MDT0001 complete [ 4087.497518] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 11:30:34 (1760455834) [ 4089.828969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4090.026705] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4090.063725] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4091.772479] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 4095.195904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4095.365227] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4097.125131] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 4100.405064] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4102.572766] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4104.969887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4105.112527] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4106.607697] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 4109.692273] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4115.427258] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4115.427321] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4115.427861] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4115.427867] Lustre: Skipped 2 previous similar messages [ 4115.431086] Lustre: Skipped 2 previous similar messages [ 4119.722242] Lustre: server umount lustre-OST0000 complete [ 4125.665810] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4125.666380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4125.672738] Lustre: Skipped 1 previous similar message [ 4125.675841] Lustre: Skipped 2 previous similar messages [ 4127.304985] Lustre: server umount lustre-MDT0000 complete [ 4128.642651] LustreError: 318817:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760455875 with bad export cookie 15858582668421967217 [ 4128.646651] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4128.647899] LustreError: 318817:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4133.681992] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 4134.876325] Key type lgssc unregistered [ 4134.979323] LNet: 322491:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4134.981497] LNet: Removed LNI 192.168.203.156@tcp [ 4135.274613] Key type .llcrypt unregistered [ 4135.275961] Key type ._llcrypt unregistered [ 4146.571433] alg: No test for adler32 (adler32-zlib) [ 4147.325330] Key type ._llcrypt registered [ 4147.326270] Key type .llcrypt registered [ 4147.360397] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_hostid [ 4152.299871] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 4152.614234] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4152.818578] Lustre: Lustre: Build Version: 2.15.7_5_g4a1d30d [ 4152.885114] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 4152.886736] LNet: Accept secure, port 988 [ 4154.456146] Key type lgssc registered [ 4154.811765] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4157.682502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4160.019791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4162.324513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4164.553178] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4166.214376] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4167.896774] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4170.130446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4170.156167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4171.251276] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4171.261590] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4171.308816] Lustre: lustre-MDT0000: new disk, initializing [ 4171.337620] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4171.344443] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4172.741186] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 4176.826275] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4176.848852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4176.882154] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4176.900858] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 4176.902414] Lustre: Skipped 1 previous similar message [ 4176.936961] Lustre: lustre-MDT0001: new disk, initializing [ 4176.968915] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4176.980954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4176.984302] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4178.512863] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 4182.323997] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4184.294518] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4186.605656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4186.630960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4186.634251] Lustre: lustre-OST2710-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4186.733521] Lustre: lustre-OST2710: new disk, initializing [ 4186.735266] Lustre: srv-lustre-OST2710: No data found on store. Initialize space: rc = -61 [ 4186.764179] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 4188.223477] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 4188.285180] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 4188.288713] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 4192.204724] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 4195.020371] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4195.056820] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4195.060289] Lustre: lustre-OST03e8-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 4195.128639] Lustre: lustre-OST03e8: new disk, initializing [ 4195.131851] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space: rc = -61 [ 4195.164882] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 4196.756886] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 4198.026605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 4198.030837] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 4199.972211] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 4203.036469] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 40 [ 4203.134022] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4204.743086] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 40 [ 4204.839807] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4206.280042] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 40 [ 4206.351612] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4207.894641] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 40 [ 4207.972525] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4212.705344] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4212.709427] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4212.715266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4213.732872] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4213.735794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4213.739765] Lustre: Skipped 2 previous similar messages [ 4213.745320] Lustre: Skipped 2 previous similar messages [ 4215.263446] Lustre: server umount lustre-MDT0000 complete [ 4216.718123] LustreError: 329535:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760455963 with bad export cookie 9042115482076628816 [ 4216.720738] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4216.720923] LustreError: 329535:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4216.855453] Lustre: server umount lustre-MDT0001 complete [ 4224.480131] Lustre: 331911:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760455965/real 1760455965] req@00000000529d8b89 x1845971805821760/t0(0) o39->lustre-MDT0001-lwp-OST2710@0@lo:12/10 lens 224/224 e 0 to 1 dl 1760455971 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4224.502319] Lustre: server umount lustre-OST2710 complete [ 4232.160289] Lustre: 332112:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760455973/real 1760455973] req@00000000529d8b89 x1845971805822208/t0(0) o39->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1760455979 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4232.246649] Lustre: server umount lustre-OST03e8 complete [ 4239.185650] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_hostid [ 4241.957219] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 4245.213799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4247.356933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4249.314803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4251.344070] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4253.578195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4253.601876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4253.679965] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4253.688705] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4253.725282] Lustre: lustre-MDT0000: new disk, initializing [ 4253.747202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4253.753095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4255.146151] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 4259.294890] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4259.316992] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4259.350556] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4259.438393] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4261.109290] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 4265.200836] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4267.098611] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4269.410598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4269.433485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4269.435810] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4269.518218] Lustre: lustre-OST0000: new disk, initializing [ 4269.519385] Lustre: Skipped 1 previous similar message [ 4269.520909] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 4269.523012] Lustre: Skipped 2 previous similar messages [ 4269.545664] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4269.547299] Lustre: Skipped 1 previous similar message [ 4270.135593] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4270.137996] Lustre: Skipped 1 previous similar message [ 4270.139426] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4270.951274] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 4274.823576] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4276.276393] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 4276.353410] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4277.859127] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 4277.945800] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4280.289497] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4280.290172] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4280.291792] LustreError: Skipped 1 previous similar message [ 4280.292351] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4280.298688] Lustre: Skipped 1 previous similar message [ 4280.303030] Lustre: Skipped 2 previous similar messages [ 4284.763984] Lustre: server umount lustre-OST0000 complete [ 4290.016782] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4290.019231] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4290.023924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4293.426946] LustreError: 337098:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760456040 with bad export cookie 9042115482076630419 [ 4293.429230] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4293.430208] LustreError: 337098:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4293.545738] Lustre: server umount lustre-MDT0001 complete [ 4293.546935] Lustre: Skipped 1 previous similar message [ 4296.348274] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 11:34:03 (1760456043) [ 4296.846106] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 4297.384008] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 11:34:04 (1760456044) [ 4298.512748] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 4301.060902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4302.359513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4303.658203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4304.991066] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4305.373629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4307.661043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4307.777819] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4307.785459] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4307.842100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4309.257440] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 4312.176203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4312.210111] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4313.789108] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 4316.442822] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4318.578982] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4320.795691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4320.823249] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4320.900135] Lustre: Denying initial registration attempt from nid 192.168.203.156@tcp, specified as failover [ 4320.904380] LustreError: 160-7: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 4320.907377] LustreError: 343025:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -99 [ 4320.909318] LustreError: 343025:0:(obd_mount_server.c:1644:server_put_super()) no obd lustre-OST0000 [ 4320.911136] LustreError: 343025:0:(obd_mount_server.c:131:server_deregister_mount()) lustre-OST0000 not registered [ 4320.971848] Lustre: server umount lustre-OST0000 complete [ 4320.972975] LustreError: 343025:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 4322.785234] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4322.787896] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4322.791565] Lustre: Skipped 1 previous similar message [ 4322.794173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4322.796782] Lustre: Skipped 1 previous similar message [ 4328.930788] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4328.937281] LustreError: Skipped 1 previous similar message [ 4330.094949] LustreError: 341068:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760456077 with bad export cookie 9042115482076631259 [ 4330.097654] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4330.100379] LustreError: 341068:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4334.861211] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 4335.844480] Key type lgssc unregistered [ 4335.949456] LNet: 344097:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4335.954580] LNet: Removed LNI 192.168.203.156@tcp [ 4336.274651] Key type .llcrypt unregistered [ 4336.275681] Key type ._llcrypt unregistered [ 4341.341735] alg: No test for adler32 (adler32-zlib) [ 4342.094379] Key type ._llcrypt registered [ 4342.095264] Key type .llcrypt registered [ 4342.130921] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 4342.449279] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4342.724892] Lustre: Lustre: Build Version: 2.15.7_5_g4a1d30d [ 4342.807567] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 4342.809710] LNet: Accept secure, port 988 [ 4344.392225] Key type lgssc registered [ 4344.801636] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4347.833664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4349.472267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4351.009598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4352.522492] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4352.939960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4359.051620] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 4362.079481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4363.173451] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4363.180153] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4363.237464] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4364.710970] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 4367.491932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4367.529623] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4367.541279] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4367.621503] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4369.216977] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 4372.214462] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4374.220808] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4376.378090] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4376.403176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4376.477257] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4376.536535] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4378.022426] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 4380.702757] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4383.203547] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4383.203642] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4383.207941] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4383.210832] Lustre: Skipped 1 previous similar message [ 4387.736986] Lustre: server umount lustre-OST0000 complete [ 4393.442334] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4393.442504] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4393.443771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4393.447938] LustreError: Skipped 1 previous similar message [ 4393.456264] Lustre: Skipped 2 previous similar messages [ 4395.161986] Lustre: server umount lustre-MDT0000 complete [ 4396.670407] LustreError: 347777:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760456143 with bad export cookie 18330512254521095769 [ 4396.673622] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4396.677261] LustreError: 347777:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4396.814153] Lustre: server umount lustre-MDT0001 complete [ 4400.621251] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 4401.594771] Key type lgssc unregistered [ 4401.709292] LNet: 350674:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4401.713461] LNet: Removed LNI 192.168.203.156@tcp [ 4402.022500] Key type .llcrypt unregistered [ 4402.023971] Key type ._llcrypt unregistered [ 4410.959782] alg: No test for adler32 (adler32-zlib) [ 4411.711328] Key type ._llcrypt registered [ 4411.712295] Key type .llcrypt registered [ 4411.752659] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 4412.183928] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4412.410510] Lustre: Lustre: Build Version: 2.15.7_5_g4a1d30d [ 4412.476996] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 4412.478531] LNet: Accept secure, port 988 [ 4414.048104] Key type lgssc registered [ 4414.421775] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4417.090093] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4418.242879] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4418.256108] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4419.586326] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 4422.319392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4422.426308] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4423.810968] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 4426.309353] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4428.262682] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4430.531583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4430.647780] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4432.057325] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 4434.647444] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4443.106528] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4443.107797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4443.111359] Lustre: Skipped 2 previous similar messages [ 4443.111676] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4443.115192] Lustre: Skipped 2 previous similar messages [ 4446.248089] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.56@tcp (stopping) [ 4448.225754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4448.227627] Lustre: Skipped 2 previous similar messages [ 4448.838301] Lustre: server umount lustre-MDT0000 complete [ 4450.175977] LustreError: 354084:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760456197 with bad export cookie 15840879551967751995 [ 4450.178392] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4450.180691] LustreError: 354084:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4451.362177] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4451.377350] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.56@tcp (stopping) [ 4453.345294] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4453.345377] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4455.393450] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4455.393560] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4455.404175] Lustre: Skipped 1 previous similar message [ 4456.391992] Lustre: server umount lustre-MDT0001 complete [ 4459.325633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4463.120558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4467.681891] Lustre: Evicted from MGS (at 192.168.203.156@tcp) after server handle changed from 0xdbd61c0620c96f3b to 0xdbd61c0620c98b18 [ 4467.690049] Lustre: MGC192.168.203.156@tcp: Connection restored to 192.168.203.156@tcp (at 0@lo) [ 4467.806477] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4467.828292] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4467.852398] Lustre: lustre-OST0000: deleting orphan objects from 0x0:166 to 0x0:193 [ 4469.353069] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 4472.460442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4472.569211] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4474.140319] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 4476.889358] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4477.924276] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 4478.477808] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 1 sec [ 4480.308592] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4480.995607] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 4485.133179] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 4 sec [ 4488.161278] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4488.163397] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4488.166965] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4493.095335] Lustre: server umount lustre-OST0000 complete [ 4496.354619] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4496.359866] Lustre: Skipped 2 previous similar messages [ 4500.686023] Lustre: server umount lustre-MDT0000 complete [ 4501.473158] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4501.476669] LustreError: Skipped 3 previous similar messages [ 4501.967995] LustreError: 354084:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760456249 with bad export cookie 15840879551967759128 [ 4501.971089] LustreError: 354084:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4501.971251] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4506.967701] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 4508.061365] Key type lgssc unregistered [ 4508.176446] LNet: 358626:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4508.178984] LNet: Removed LNI 192.168.203.156@tcp [ 4508.485307] Key type .llcrypt unregistered [ 4508.486381] Key type ._llcrypt unregistered [ 4517.736195] alg: No test for adler32 (adler32-zlib) [ 4518.487446] Key type ._llcrypt registered [ 4518.488311] Key type .llcrypt registered [ 4518.526510] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 4518.954229] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4519.209107] Lustre: Lustre: Build Version: 2.15.7_5_g4a1d30d [ 4519.292386] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 4519.294380] LNet: Accept secure, port 988 [ 4520.880114] Key type lgssc registered [ 4521.277108] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4524.023759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4525.165426] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4525.180046] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4526.519410] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 4529.631469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4529.739065] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4531.150037] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 4533.765960] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4535.881252] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4539.874782] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4539.874899] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4539.875854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4539.882401] Lustre: Skipped 1 previous similar message [ 4543.110374] Lustre: server umount lustre-MDT0000 complete [ 4544.393282] LustreError: 360068:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760456291 with bad export cookie 4651207520951968395 [ 4544.396397] LustreError: 360068:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4544.397141] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4544.512143] Lustre: server umount lustre-MDT0001 complete [ 4549.385156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4549.478116] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4549.488887] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4549.556261] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4550.973635] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 4553.786712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4553.816977] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4553.827497] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4553.918960] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4555.342203] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 4558.001966] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4559.918354] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4562.451138] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4562.519249] LustreError: 13b-9: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 4562.522828] LustreError: 160-7: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 4562.526282] LustreError: 364778:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -2 [ 4562.528463] LustreError: 364778:0:(obd_mount_server.c:1644:server_put_super()) no obd lustre-OST0000 [ 4562.531278] LustreError: 364778:0:(obd_mount_server.c:131:server_deregister_mount()) lustre-OST0000 not registered [ 4562.586293] Lustre: server umount lustre-OST0000 complete [ 4562.587571] LustreError: 364778:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4564.820241] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4564.896558] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4564.952205] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4566.444610] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 4567.698691] Lustre: lustre-OST0000: deleting orphan objects from 0x0:166 to 0x0:225 [ 4568.884640] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4571.449935] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4571.481094] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4571.485268] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 4571.515621] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4571.551807] Lustre: lustre-OST0001: new disk, initializing [ 4571.553604] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 4573.069135] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 4575.541582] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4576.533190] Lustre: server umount lustre-OST0001 complete [ 4577.783380] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4578.272882] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4578.275426] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4578.279758] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4578.281700] Lustre: Skipped 2 previous similar messages [ 4582.882615] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4582.883421] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4582.884212] Lustre: Skipped 1 previous similar message [ 4582.890641] LustreError: Skipped 2 previous similar messages [ 4588.003482] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4588.004405] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4588.005318] Lustre: Skipped 1 previous similar message [ 4588.008247] LustreError: Skipped 1 previous similar message [ 4592.096104] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 4592.164292] Lustre: server umount lustre-OST0000 complete [ 4595.168913] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4595.171895] LustreError: Skipped 1 previous similar message [ 4595.173599] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4595.177355] Lustre: Skipped 1 previous similar message [ 4595.178757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4595.180589] Lustre: Skipped 2 previous similar messages [ 4600.782160] LustreError: 365162:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760456347 with bad export cookie 4651207520951968934 [ 4600.786066] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4600.787972] LustreError: 365162:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4600.918432] Lustre: server umount lustre-MDT0001 complete [ 4600.919635] Lustre: Skipped 1 previous similar message [ 4604.451633] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 4605.464615] Key type lgssc unregistered [ 4605.567339] LNet: 368145:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4605.569938] LNet: Removed LNI 192.168.203.156@tcp [ 4605.869565] Key type .llcrypt unregistered [ 4605.870526] Key type ._llcrypt unregistered [ 4617.363652] alg: No test for adler32 (adler32-zlib) [ 4618.115341] Key type ._llcrypt registered [ 4618.116288] Key type .llcrypt registered [ 4618.158386] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_hostid [ 4624.342593] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 4624.871751] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4625.073354] Lustre: Lustre: Build Version: 2.15.7_5_g4a1d30d [ 4625.154711] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 4625.157348] LNet: Accept secure, port 988 [ 4626.744084] Key type lgssc registered [ 4627.075957] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4630.403772] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4632.682506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4634.767639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4636.876263] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4639.094857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4639.115356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4640.203021] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4640.212182] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4640.251651] Lustre: lustre-MDT0000: new disk, initializing [ 4640.275684] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4640.282815] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4641.759543] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 4645.958672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4645.983733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4646.013297] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4646.029709] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 4646.031925] Lustre: Skipped 1 previous similar message [ 4646.065127] Lustre: lustre-MDT0001: new disk, initializing [ 4646.092604] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4646.103102] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4646.105664] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4647.574188] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 4651.514584] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4653.460109] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4655.784859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4655.807982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4655.812026] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4655.913229] Lustre: lustre-OST0000: new disk, initializing [ 4655.915181] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 4655.937570] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4657.288123] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 4661.399792] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4663.085956] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 4664.820463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4664.823301] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4665.210925] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 4666.610996] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 4666.693749] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4669.922587] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4669.924033] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4669.924860] LustreError: Skipped 1 previous similar message [ 4669.929225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4669.931303] Lustre: Skipped 1 previous similar message [ 4673.442322] Lustre: server umount lustre-OST0000 complete [ 4675.041150] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4675.041768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4675.044937] Lustre: Skipped 2 previous similar messages [ 4675.050305] Lustre: Skipped 1 previous similar message [ 4680.160889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4680.163989] Lustre: Skipped 1 previous similar message [ 4685.281339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4685.284129] Lustre: Skipped 1 previous similar message [ 4689.376128] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 4689.412996] Lustre: server umount lustre-MDT0000 complete [ 4690.894073] LustreError: 372226:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760456438 with bad export cookie 1493381617671870033 [ 4690.897062] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4690.897964] LustreError: 372226:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4691.032487] Lustre: server umount lustre-MDT0001 complete [ 4693.808819] Lustre: DEBUG MARKER: == conf-sanity test 60: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 11:40:41 (1760456441) [ 4695.920624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4698.177301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4705.555251] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_hostid [ 4708.633479] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 4712.142061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4714.692209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4716.939708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4718.964426] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4721.537824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4721.558494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4721.640065] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4721.650276] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4721.689503] Lustre: lustre-MDT0000: new disk, initializing [ 4721.717682] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4721.724635] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4723.158291] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 4727.595127] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4727.628383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4727.664042] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4727.680481] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 4727.682185] Lustre: Skipped 1 previous similar message [ 4727.715511] Lustre: lustre-MDT0001: new disk, initializing [ 4727.742091] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4727.751414] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4727.753830] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4729.124375] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 4732.985398] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4735.179121] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4737.741250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4737.768804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4737.770852] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4737.861973] Lustre: lustre-OST0000: new disk, initializing [ 4737.863731] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 4737.891012] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4739.211952] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 4743.150456] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4744.690197] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 4746.746353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4746.750315] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4747.853173] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 4749.197838] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 4749.272608] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4751.842415] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4751.843150] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4751.845686] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4751.847375] LustreError: Skipped 1 previous similar message [ 4751.851225] Lustre: Skipped 3 previous similar messages [ 4756.058618] Lustre: server umount lustre-OST0000 complete [ 4758.496746] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4758.500383] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4758.505975] Lustre: Skipped 1 previous similar message [ 4762.080918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4762.082718] Lustre: Skipped 2 previous similar messages [ 4763.392973] Lustre: server umount lustre-MDT0000 complete [ 4764.555059] LustreError: 380001:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760456511 with bad export cookie 1493381617671870873 [ 4764.557295] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4764.559159] LustreError: 380001:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4767.431663] Lustre: DEBUG MARKER: == conf-sanity test 61: large xattr ====================== 11:41:54 (1760456514) [ 4770.205032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4770.347587] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4770.367035] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4771.705683] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 4774.575857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4776.041861] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 4778.690411] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4780.650842] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4782.890375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4784.324859] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 4786.720607] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4795.256699] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61.conf-sanity [ 4795.812575] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61.conf-sanity [ 4796.377853] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61.conf-sanity [ 4796.961288] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61.conf-sanity after remounting MDS [ 4797.615371] Lustre: Failing over lustre-MDT0000 [ 4797.809308] Lustre: server umount lustre-MDT0000 complete [ 4797.810724] Lustre: Skipped 1 previous similar message [ 4798.503325] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4799.458245] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4799.461871] Lustre: Skipped 3 previous similar messages [ 4803.616218] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4803.620614] LustreError: Skipped 3 previous similar messages [ 4805.674476] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4805.679611] LustreError: Skipped 3 previous similar messages [ 4806.624185] Lustre: 370461:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760456546/real 1760456546] req@00000000430c37c9 x1845972300767104/t0(0) o400->MGC192.168.203.156@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1760456553 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4806.629695] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4810.783816] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4810.787600] LustreError: Skipped 3 previous similar messages [ 4810.920418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4812.771650] Lustre: Evicted from MGS (at 192.168.203.156@tcp) after server handle changed from 0x14b98eab436e1ce1 to 0x14b98eab436e20bc [ 4812.776918] Lustre: MGC192.168.203.156@tcp: Connection restored to 192.168.203.156@tcp (at 0@lo) [ 4812.855718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4812.857296] Lustre: Skipped 2 previous similar messages [ 4812.886782] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4814.159595] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 4816.928206] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4817.894216] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 4817.900106] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4817.916422] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 4819.663065] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4820.160872] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4821.943763] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61.conf-sanity [ 4822.549163] Lustre: Failing over lustre-MDT0000 [ 4822.730694] Lustre: server umount lustre-MDT0000 complete [ 4823.011367] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4823.011912] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4823.012248] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4823.012255] LustreError: Skipped 7 previous similar messages [ 4823.014180] Lustre: Skipped 2 previous similar messages [ 4830.176167] Lustre: 370461:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760456570/real 1760456570] req@00000000fa72f540 x1845972300774464/t0(0) o400->MGC192.168.203.156@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1760456577 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4830.181331] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4835.636206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4836.322368] Lustre: Evicted from MGS (at 192.168.203.156@tcp) after server handle changed from 0x14b98eab436e20bc to 0x14b98eab436e23be [ 4836.328618] LustreError: 389287:0:(ldlm_resource.c:1127:ldlm_resource_complain()) MGC192.168.203.156@tcp: namespace resource [0x65727473756c:0x5:0x0].0x0 (000000004e333dee) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 4836.332628] Lustre: MGC192.168.203.156@tcp: Connection restored to (at 0@lo) [ 4836.335226] Lustre: Skipped 2 previous similar messages [ 4836.424502] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4837.727821] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 4838.431798] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4841.449307] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 4841.454389] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 4841.474496] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:97 [ 4843.233710] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4843.739440] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4846.562815] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4846.563258] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4846.564790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4846.570036] Lustre: Skipped 1 previous similar message [ 4853.202060] LustreError: 385462:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760456600 with bad export cookie 1493381617671873470 [ 4853.203588] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4853.205983] LustreError: 385462:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4859.872138] Lustre: 390454:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760456601/real 1760456601] req@00000000952e353e x1845972300783360/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1760456607 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4859.949491] Lustre: server umount lustre-OST0000 complete [ 4859.950975] Lustre: Skipped 2 previous similar messages [ 4867.145744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4867.340396] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4867.345539] LustreError: Skipped 17 previous similar messages [ 4869.068068] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 4872.791421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4874.924712] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 4878.653062] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4881.490549] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4884.816386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4884.946880] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4884.951782] Lustre: Skipped 3 previous similar messages [ 4886.975462] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 4890.728875] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4897.259574] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:129 [ 4903.393101] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4903.398298] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4903.404445] Lustre: Skipped 1 previous similar message [ 4903.408469] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4903.412462] Lustre: Skipped 5 previous similar messages [ 4917.216104] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 4925.153103] Lustre: server umount lustre-MDT0000 complete [ 4925.155379] Lustre: Skipped 1 previous similar message [ 4926.618817] LustreError: 391260:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760456673 with bad export cookie 1493381617671874177 [ 4926.623272] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4926.625269] LustreError: 391260:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4932.343332] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 4933.426177] Key type lgssc unregistered [ 4933.553660] LNet: 394959:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4933.557832] LNet: Removed LNI 192.168.203.156@tcp [ 4933.942726] Key type .llcrypt unregistered [ 4933.943940] Key type ._llcrypt unregistered [ 4945.359194] alg: No test for adler32 (adler32-zlib) [ 4946.110272] Key type ._llcrypt registered [ 4946.111097] Key type .llcrypt registered [ 4946.148713] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 4946.463624] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4946.680462] Lustre: Lustre: Build Version: 2.15.7_5_g4a1d30d [ 4946.754318] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 4946.756339] LNet: Accept secure, port 988 [ 4948.336138] Key type lgssc registered [ 4948.819797] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4952.401964] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4952.408281] LustreError: 396442:0:(osd_handler.c:8211:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 4952.413857] LustreError: 396442:0:(obd_config.c:774:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 4952.419330] LustreError: 396442:0:(obd_mount.c:200:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 4952.423837] LustreError: 396442:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 4952.428892] LustreError: 396442:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 4955.767922] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4955.771063] LustreError: 396839:0:(osd_handler.c:8211:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 4955.775594] LustreError: 396839:0:(obd_config.c:774:class_setup()) setup lustre-OST0000-osd failed (-22) [ 4955.778099] LustreError: 396839:0:(obd_mount.c:200:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 4955.780563] LustreError: 396839:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 4955.783502] LustreError: 396839:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 4962.498576] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 4963.445966] Key type lgssc unregistered [ 4963.553339] LNet: 397795:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4963.555861] LNet: Removed LNI 192.168.203.156@tcp [ 4963.842843] Key type .llcrypt unregistered [ 4963.843738] Key type ._llcrypt unregistered [ 4970.909181] alg: No test for adler32 (adler32-zlib) [ 4971.661338] Key type ._llcrypt registered [ 4971.662237] Key type .llcrypt registered [ 4971.707701] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_hostid [ 4977.806661] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 4978.282637] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4978.477693] Lustre: Lustre: Build Version: 2.15.7_5_g4a1d30d [ 4978.567041] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 4978.568523] LNet: Accept secure, port 988 [ 4980.160223] Key type lgssc registered [ 4980.593643] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4984.492534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4986.895340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4989.475763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4991.923598] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4994.631050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4994.655929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4995.762168] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4995.774451] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4995.819454] Lustre: lustre-MDT0000: new disk, initializing [ 4995.843188] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4995.849594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4997.419706] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 5002.112364] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5002.147037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5002.188251] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5002.210262] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 5002.212127] Lustre: Skipped 1 previous similar message [ 5002.254195] Lustre: lustre-MDT0001: new disk, initializing [ 5002.287507] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5002.304191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5002.310251] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5003.952067] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 5007.690122] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5009.508528] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5012.016448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5012.037864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5012.040081] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 5012.134774] Lustre: lustre-OST0000: new disk, initializing [ 5012.136474] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5012.162463] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5013.566130] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5013.568734] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5013.644773] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 5017.319191] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5018.802392] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 5018.900283] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5020.541814] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 5020.645980] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5023.714912] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5023.716332] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5023.720456] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5023.722285] Lustre: Skipped 1 previous similar message [ 5023.727941] Lustre: Skipped 1 previous similar message [ 5027.522494] Lustre: server umount lustre-OST0000 complete [ 5032.929263] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5032.932947] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5032.936313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5033.953664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5033.954638] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5035.146059] Lustre: server umount lustre-MDT0000 complete [ 5036.581408] LustreError: 401060:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760456783 with bad export cookie 7243192111794684560 [ 5036.588114] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5036.593768] LustreError: 401060:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5036.748216] Lustre: server umount lustre-MDT0001 complete [ 5039.670927] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 5040.271846] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 11:46:27 (1760456787) [ 5042.865147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5043.020124] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5043.047741] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5044.473546] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 5047.842103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5047.970960] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5049.663783] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 5052.576617] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5054.522643] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5057.086845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5057.234963] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5058.824510] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 5061.958706] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5064.685847] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5064.716703] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5064.718988] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 5064.769373] Lustre: lustre-OST0001: new disk, initializing [ 5064.770812] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 5066.048535] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5066.051217] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5066.332448] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 5070.689666] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5076.450170] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5076.452931] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5076.454812] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5076.458316] Lustre: Skipped 1 previous similar message [ 5076.462755] Lustre: Skipped 1 previous similar message [ 5078.762464] Lustre: server umount lustre-OST0001 complete [ 5081.571024] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5081.575161] LustreError: Skipped 1 previous similar message [ 5086.690974] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5086.694879] LustreError: Skipped 1 previous similar message [ 5091.813487] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5091.814427] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5091.817726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5091.817734] Lustre: Skipped 1 previous similar message [ 5091.821063] Lustre: Skipped 1 previous similar message [ 5091.828356] LustreError: Skipped 1 previous similar message [ 5096.932756] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5096.940352] LustreError: Skipped 1 previous similar message [ 5102.051756] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5102.053366] Lustre: Skipped 4 previous similar messages [ 5105.120120] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 5105.196946] Lustre: server umount lustre-OST0000 complete [ 5107.170553] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5114.119853] LustreError: 405646:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760456861 with bad export cookie 7243192111794685400 [ 5114.124715] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5114.124821] LustreError: 405646:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5120.334482] Lustre: server umount lustre-MDT0001 complete [ 5120.336724] Lustre: Skipped 1 previous similar message [ 5125.236920] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 5126.431530] Key type lgssc unregistered [ 5126.567490] LNet: 410564:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5126.571505] LNet: Removed LNI 192.168.203.156@tcp [ 5126.966886] Key type .llcrypt unregistered [ 5126.967875] Key type ._llcrypt unregistered [ 5139.576414] alg: No test for adler32 (adler32-zlib) [ 5140.327404] Key type ._llcrypt registered [ 5140.328401] Key type .llcrypt registered [ 5140.363597] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_hostid [ 5146.494124] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 5147.072970] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5147.225114] Lustre: Lustre: Build Version: 2.15.7_5_g4a1d30d [ 5147.303390] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 5147.305172] LNet: Accept secure, port 988 [ 5148.888113] Key type lgssc registered [ 5149.228527] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5152.625466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5155.188488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5157.396058] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5159.558653] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5161.749144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5161.781773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5162.881980] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5162.893031] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5162.935074] Lustre: lustre-MDT0000: new disk, initializing [ 5162.967620] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5162.975169] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5164.618004] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 5169.196649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5169.221913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5169.256965] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5169.279878] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 5169.282210] Lustre: Skipped 1 previous similar message [ 5169.323060] Lustre: lustre-MDT0001: new disk, initializing [ 5169.353193] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5169.366297] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5169.369141] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5170.857159] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 5174.734458] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5176.836751] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5179.425453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5179.455025] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5179.459996] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 5179.567811] Lustre: lustre-OST0000: new disk, initializing [ 5179.570203] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5179.601951] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5181.178319] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 5185.733877] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5187.273176] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 5188.597110] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5188.599563] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5189.383161] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 5190.851844] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 5190.917241] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5193.697975] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5193.699135] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5193.700780] LustreError: Skipped 1 previous similar message [ 5193.705369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5193.707888] Lustre: Skipped 1 previous similar message [ 5197.801015] Lustre: server umount lustre-OST0000 complete [ 5200.353763] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5200.358390] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5200.364202] Lustre: Skipped 1 previous similar message [ 5200.367174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5203.938182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5203.938483] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5205.651132] Lustre: server umount lustre-MDT0000 complete [ 5207.041954] LustreError: 415505:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760456954 with bad export cookie 17583315828091608044 [ 5207.044225] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5207.044900] LustreError: 415505:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5207.175297] Lustre: server umount lustre-MDT0001 complete [ 5210.490908] Lustre: DEBUG MARKER: == conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================================================== 11:49:17 (1760456957) [ 5214.197567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5217.767036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5217.967278] Lustre: lustre-MDT0000: new disk, initializing [ 5217.977813] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5217.999262] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5219.533196] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 5222.931503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5223.080976] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5225.027957] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 5228.216480] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5230.961150] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5233.635222] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5233.635411] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5233.636427] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5233.636435] Lustre: Skipped 1 previous similar message [ 5233.654321] Lustre: Skipped 1 previous similar message [ 5237.976177] Lustre: server umount lustre-MDT0000 complete [ 5238.752967] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5238.756479] LustreError: Skipped 1 previous similar message [ 5239.365142] LustreError: 419793:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760456986 with bad export cookie 17583315828091608884 [ 5239.368302] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5239.369154] LustreError: 419793:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5243.290379] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 11:49:50 (1760456990) [ 5246.011279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5246.201924] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5246.223418] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5247.458524] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 5250.889984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5252.442481] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 5255.914810] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5258.559128] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5260.625926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5262.435495] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 5265.281266] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5269.582880] Lustre: Permanently deactivating lustre-OST0000 [ 5269.591407] Lustre: Setting parameter lustre-OST0000-osc.osc.active in log lustre-client [ 5275.635455] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5277.202724] LustreError: 425505:0:(mgs_llog.c:1551:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002020 [ 5277.208806] LustreError: 425505:0:(mgs_llog.c:1605:mgs_replace_nids()) Only MGS is allowed to be started [ 5277.213067] LustreError: 425505:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -115 [ 5277.964761] LustreError: 425553:0:(mgs_llog.c:1605:mgs_replace_nids()) Only MGS is allowed to be started [ 5277.967633] LustreError: 425553:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -115 [ 5284.831428] Lustre: server umount lustre-OST0000 complete [ 5284.832753] Lustre: Skipped 1 previous similar message [ 5285.734098] LustreError: 425802:0:(mgs_llog.c:1605:mgs_replace_nids()) Only MGS is allowed to be started [ 5285.736981] LustreError: 425802:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -115 [ 5286.881627] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5286.886967] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5286.895513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5286.898804] Lustre: Skipped 1 previous similar message [ 5292.512153] Lustre: 425899:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760457033/real 1760457033] req@000000003e5b2038 x1845972848119104/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1760457039 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 5293.537703] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5293.548812] LustreError: Skipped 1 previous similar message [ 5294.144435] LustreError: 424360:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760457041 with bad export cookie 17583315828091609423 [ 5294.148155] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5294.149504] LustreError: 424360:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5300.192214] Lustre: 426100:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760457041/real 1760457041] req@000000005f7b48d3 x1845972848120128/t0(0) o9->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1760457047 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 5300.360145] Lustre: server umount lustre-MDT0001 complete [ 5300.362373] Lustre: Skipped 1 previous similar message [ 5303.813160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5305.631480] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 5306.574493] LustreError: 427008:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -6 [ 5308.535763] Lustre: 427353:0:(mgs_llog.c:1311:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.156@tcp [ 5315.552153] Lustre: 427452:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760457056/real 1760457056] req@000000003abe05bd x1845972848120576/t0(0) o251->MGC192.168.203.156@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1760457062 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 5320.437821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5320.634038] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5320.638386] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5320.642427] Lustre: Skipped 1 previous similar message [ 5320.667055] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5320.669623] Lustre: Skipped 2 previous similar messages [ 5322.137038] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 5325.652979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5325.791910] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5327.776672] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 5331.137186] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5333.427035] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5334.058325] Lustre: Permanently reactivating lustre-OST0000 [ 5334.062522] Lustre: Modifying parameter lustre-OST0000-osc.osc.active in log lustre-client [ 5334.064172] Lustre: Skipped 2 previous similar messages [ 5347.404875] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5349.365756] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 5352.337159] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5364.196843] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5364.197314] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5364.203799] Lustre: Skipped 1 previous similar message [ 5364.205184] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5364.214759] Lustre: Skipped 3 previous similar messages [ 5370.277982] Lustre: server umount lustre-OST0000 complete [ 5370.279462] Lustre: Skipped 1 previous similar message [ 5379.224998] LustreError: 428866:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760457126 with bad export cookie 17583315828091610368 [ 5379.230351] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5379.231493] LustreError: 428866:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5384.261183] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 5385.253354] Key type lgssc unregistered [ 5385.359344] LNet: 432297:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5385.361434] LNet: Removed LNI 192.168.203.156@tcp [ 5385.734600] Key type .llcrypt unregistered [ 5385.736671] Key type ._llcrypt unregistered [ 5393.305903] alg: No test for adler32 (adler32-zlib) [ 5394.058366] Key type ._llcrypt registered [ 5394.059233] Key type .llcrypt registered [ 5394.094823] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_hostid [ 5400.262792] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 5400.527947] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5400.747089] Lustre: Lustre: Build Version: 2.15.7_5_g4a1d30d [ 5400.811959] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 5400.813339] LNet: Accept secure, port 988 [ 5402.384082] Key type lgssc registered [ 5402.686622] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5405.423704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5407.349162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5409.400677] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5412.219541] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5415.265891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5415.305042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5416.401202] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5416.413000] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5416.451502] Lustre: lustre-MDT0000: new disk, initializing [ 5416.478186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5416.484894] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5418.198275] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 5423.020504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5423.051280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5423.087064] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5423.103602] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 5423.105325] Lustre: Skipped 1 previous similar message [ 5423.142184] Lustre: lustre-MDT0001: new disk, initializing [ 5423.173891] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5423.186041] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5423.188676] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5424.783670] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 5429.338403] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5431.333916] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5433.564729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5433.592376] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5433.594357] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 5433.681965] Lustre: lustre-OST0000: new disk, initializing [ 5433.683400] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5433.706711] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5435.066495] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 5438.989895] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5438.996633] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5439.224209] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5441.297930] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 5441.383972] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5443.137344] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 5443.220800] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5444.068661] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5444.068936] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5444.070839] LustreError: Skipped 1 previous similar message [ 5444.074876] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5444.078057] Lustre: Skipped 1 previous similar message [ 5445.972368] Lustre: server umount lustre-OST0000 complete [ 5448.673460] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5448.678313] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5448.685944] Lustre: Skipped 1 previous similar message [ 5448.689305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5453.800199] Lustre: server umount lustre-MDT0000 complete [ 5454.306836] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5454.314373] LustreError: Skipped 1 previous similar message [ 5455.238468] LustreError: 435561:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760457202 with bad export cookie 16931208640250077002 [ 5455.242504] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5455.243247] LustreError: 435561:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5455.379273] Lustre: server umount lustre-MDT0001 complete [ 5458.874936] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 5459.694386] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 11:53:26 (1760457206) [ 5462.625273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5462.792601] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5462.818562] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5464.707358] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 5468.130296] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5468.748074] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5468.955097] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5470.664725] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 5473.986411] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5476.665234] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5479.976434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5480.118445] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5481.915532] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 5484.674075] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5485.561789] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 5496.489040] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 5498.852970] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5498.854300] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5498.854972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5498.854978] Lustre: Skipped 2 previous similar messages [ 5498.859127] Lustre: Skipped 2 previous similar messages [ 5503.972698] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5503.976480] Lustre: Skipped 2 previous similar messages [ 5504.455319] Lustre: server umount lustre-OST0000 complete [ 5509.091503] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5509.092272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5509.098682] Lustre: Skipped 1 previous similar message [ 5512.485205] Lustre: server umount lustre-MDT0000 complete [ 5514.185832] LustreError: 440147:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760457261 with bad export cookie 16931208640250077856 [ 5514.189859] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5514.191995] LustreError: 440147:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5519.785910] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 5521.200219] Key type lgssc unregistered [ 5521.348611] LNet: 444031:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5521.352423] LNet: Removed LNI 192.168.203.156@tcp [ 5521.779467] Key type .llcrypt unregistered [ 5521.780969] Key type ._llcrypt unregistered [ 5532.255406] alg: No test for adler32 (adler32-zlib) [ 5533.007307] Key type ._llcrypt registered [ 5533.008181] Key type .llcrypt registered [ 5533.052531] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 5533.182813] Key type .llcrypt unregistered [ 5533.184899] Key type ._llcrypt unregistered [ 5540.644500] alg: No test for adler32 (adler32-zlib) [ 5541.396339] Key type ._llcrypt registered [ 5541.397152] Key type .llcrypt registered [ 5541.438280] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 5541.839328] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5541.977627] Lustre: Lustre: Build Version: 2.15.7_5_g4a1d30d [ 5542.043071] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 5542.044398] LNet: Accept secure, port 988 [ 5543.616097] Key type lgssc registered [ 5544.154837] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5547.859682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5548.989226] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5549.002635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5550.985273] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 5554.145990] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5554.689959] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5554.826265] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5556.487443] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 5560.010607] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5560.290821] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5560.298134] LustreError: Skipped 2 previous similar messages [ 5563.009755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5563.127444] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5564.135753] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 5564.541342] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 5568.481382] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5568.484320] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5568.490382] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5569.507612] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5569.508953] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5569.520369] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5570.036520] Lustre: server umount lustre-OST0000 complete [ 5573.091724] LustreError: 448527:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760457320 with bad export cookie 3138441998444142507 [ 5573.093150] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 5573.098555] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5573.109135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5573.112385] Lustre: Skipped 1 previous similar message [ 5573.839846] Lustre: server umount lustre-MDT0000 complete [ 5574.625571] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5574.629120] LustreError: Skipped 1 previous similar message [ 5575.538702] LustreError: 448527:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760457322 with bad export cookie 3138441998444142003 [ 5575.544724] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5575.545410] LustreError: 448527:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5578.887952] Lustre: server umount lustre-MDT0001 complete [ 5584.332612] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 5585.418115] Key type lgssc unregistered [ 5585.525315] LNet: 449619:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5585.527658] LNet: Removed LNI 192.168.203.156@tcp [ 5585.886266] Key type .llcrypt unregistered [ 5585.888381] Key type ._llcrypt unregistered [ 5597.227966] alg: No test for adler32 (adler32-zlib) [ 5597.980278] Key type ._llcrypt registered [ 5597.981665] Key type .llcrypt registered [ 5598.036144] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 5598.440577] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5598.709466] Lustre: Lustre: Build Version: 2.15.7_5_g4a1d30d [ 5598.795675] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 5598.797225] LNet: Accept secure, port 988 [ 5600.376073] Key type lgssc registered [ 5600.668171] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5603.242496] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5620.768329] LustreError: 451050:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.203.156@tcp: failed processing log, type 1: rc = -5 [ 5651.488238] LustreError: 451050:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.203.156@tcp: failed processing log, type 4: rc = -110 [ 5681.184175] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5683.470397] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 5686.877162] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5690.078653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5703.822486] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5703.860635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5703.896373] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:97 [ 5705.722504] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 5709.283583] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5709.560704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5709.753378] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5711.696486] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 5715.367952] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5718.346147] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5724.641651] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5724.647092] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5724.655686] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5725.154911] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5725.159711] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5725.167271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5726.180418] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5726.185382] Lustre: Skipped 1 previous similar message [ 5727.308875] Lustre: server umount lustre-OST0000 complete [ 5730.273567] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5730.278716] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5730.286966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5735.202570] Lustre: server umount lustre-MDT0000 complete [ 5736.418600] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5736.426945] LustreError: Skipped 1 previous similar message [ 5737.094259] LustreError: 453943:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760457484 with bad export cookie 16546003118546598043 [ 5737.101205] LustreError: 453943:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5737.101414] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5737.308339] Lustre: server umount lustre-MDT0001 complete [ 5743.138550] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 5744.460636] Key type lgssc unregistered [ 5744.599368] LNet: 454746:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5744.604165] LNet: Removed LNI 192.168.203.156@tcp [ 5745.061837] Key type .llcrypt unregistered [ 5745.063748] Key type ._llcrypt unregistered [ 5755.570495] alg: No test for adler32 (adler32-zlib) [ 5756.323307] Key type ._llcrypt registered [ 5756.324219] Key type .llcrypt registered [ 5756.368188] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 5756.673030] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5756.898664] Lustre: Lustre: Build Version: 2.15.7_5_g4a1d30d [ 5756.976864] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 5756.978202] LNet: Accept secure, port 988 [ 5758.552124] Key type lgssc registered [ 5759.092946] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5762.907427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5764.077993] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5764.091177] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5766.068434] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 5769.186518] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5770.235143] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5770.430784] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5772.453764] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 5776.332193] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5779.308311] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5782.624261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5782.790133] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5784.786260] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 5788.324284] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5794.094262] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:129 [ 5801.441358] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5801.446056] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5801.453503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5804.076414] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.56@tcp (stopping) [ 5804.516804] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5804.524799] Lustre: Skipped 1 previous similar message [ 5806.437375] Lustre: server umount lustre-MDT0000 complete [ 5807.604680] LustreError: 456994:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.56@tcp arrived at 1760457554 with bad export cookie 2213874120595965105 [ 5807.613729] LustreError: 456994:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5808.566368] LustreError: 456994:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760457555 with bad export cookie 2213874120595964468 [ 5808.575244] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5808.575615] LustreError: 456994:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5809.633819] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5809.634198] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5809.641396] LustreError: Skipped 1 previous similar message [ 5809.649190] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5809.654568] Lustre: Skipped 3 previous similar messages [ 5813.665690] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5813.665882] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5813.673158] LustreError: Skipped 1 previous similar message [ 5814.717389] Lustre: server umount lustre-OST0000 complete [ 5818.073351] Lustre: server umount lustre-MDT0001 complete [ 5824.319160] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 5825.750563] Key type lgssc unregistered [ 5825.879433] LNet: 460035:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5825.885471] LNet: Removed LNI 192.168.203.156@tcp [ 5826.328868] Key type .llcrypt unregistered [ 5826.330985] Key type ._llcrypt unregistered [ 5838.278396] alg: No test for adler32 (adler32-zlib) [ 5839.030320] Key type ._llcrypt registered [ 5839.031202] Key type .llcrypt registered [ 5839.074908] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 5839.453768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5839.607713] Lustre: Lustre: Build Version: 2.15.7_5_g4a1d30d [ 5839.672223] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 5839.673511] LNet: Accept secure, port 988 [ 5841.256113] Key type lgssc registered [ 5841.794298] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5845.680174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5846.881364] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5846.900560] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5848.881463] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 5852.129963] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5852.932071] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5853.105813] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5855.370843] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 5859.211211] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5861.990197] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5865.266245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5865.416775] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5867.266437] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 5870.726203] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5877.738723] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:161 [ 5882.850845] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5882.858484] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5882.862031] Lustre: Skipped 1 previous similar message [ 5886.508684] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.56@tcp (stopping) [ 5887.970260] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5887.974169] Lustre: Skipped 1 previous similar message [ 5888.790069] Lustre: server umount lustre-MDT0001 complete [ 5893.089791] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5893.090609] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5893.091491] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5893.091500] Lustre: Skipped 1 previous similar message [ 5893.092194] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5897.214141] Lustre: server umount lustre-OST0000 complete [ 5898.210123] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5905.376158] Lustre: 464570:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760457646/real 1760457646] req@000000008739828b x1845973573711232/t0(0) o251->MGC192.168.203.156@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1760457652 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 5905.485674] Lustre: server umount lustre-MDT0000 complete [ 5912.810804] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 5914.131698] Key type lgssc unregistered [ 5914.268596] LNet: 465326:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5914.274207] LNet: Removed LNI 192.168.203.156@tcp [ 5914.690773] Key type .llcrypt unregistered [ 5914.692632] Key type ._llcrypt unregistered [ 5927.219786] alg: No test for adler32 (adler32-zlib) [ 5927.971340] Key type ._llcrypt registered [ 5927.972711] Key type .llcrypt registered [ 5928.014491] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_hostid [ 5934.880289] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 5935.270593] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5935.414416] Lustre: Lustre: Build Version: 2.15.7_5_g4a1d30d [ 5935.487469] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 5935.488818] LNet: Accept secure, port 988 [ 5937.064140] Key type lgssc registered [ 5937.599794] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5941.828767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5945.146750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5948.178245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5951.113965] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5955.217975] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 5959.152980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5962.408768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5962.445096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5963.542821] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5963.553732] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5963.597478] Lustre: lustre-MDT0000: new disk, initializing [ 5963.626459] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5963.637303] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5965.717869] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 5970.179542] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5973.334195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5973.362098] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5973.366623] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 5973.511309] Lustre: lustre-OST0000: new disk, initializing [ 5973.514741] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5973.518363] Lustre: Skipped 1 previous similar message [ 5973.555753] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5975.685920] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 5980.976645] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5985.595186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5988.881568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5988.916634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5988.974975] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5989.008966] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 5989.060087] Lustre: lustre-MDT0001: new disk, initializing [ 5989.083390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 5989.087536] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 5989.108120] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5991.232595] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 5996.174187] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 40 [ 5996.305078] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 5998.311885] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 40 [ 5998.440074] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6002.660623] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6002.660740] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6002.666186] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6002.681039] Lustre: Skipped 1 previous similar message [ 6002.686868] Lustre: Skipped 1 previous similar message [ 6006.865484] Lustre: server umount lustre-MDT0001 complete [ 6007.779989] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6007.794146] LustreError: Skipped 1 previous similar message [ 6012.899455] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6012.906756] LustreError: Skipped 2 previous similar messages [ 6018.017899] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6018.021215] LustreError: Skipped 1 previous similar message [ 6023.138462] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6023.138854] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6023.157529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6023.163763] Lustre: Skipped 1 previous similar message [ 6025.776844] Lustre: server umount lustre-OST0000 complete [ 6040.032191] Lustre: 473736:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760457781/real 1760457781] req@000000009f6140a4 x1845973674377088/t0(0) o251->MGC192.168.203.156@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1760457787 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 6040.128204] Lustre: server umount lustre-MDT0000 complete [ 6047.553836] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 6048.604471] Key type lgssc unregistered [ 6048.700257] LNet: 474491:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6048.702450] LNet: Removed LNI 192.168.203.156@tcp [ 6048.935666] Key type .llcrypt unregistered [ 6048.936589] Key type ._llcrypt unregistered [ 6068.577460] alg: No test for adler32 (adler32-zlib) [ 6069.330306] Key type ._llcrypt registered [ 6069.331205] Key type .llcrypt registered [ 6069.386286] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 6069.503309] Key type .llcrypt unregistered [ 6069.504805] Key type ._llcrypt unregistered [ 6077.739435] alg: No test for adler32 (adler32-zlib) [ 6078.491324] Key type ._llcrypt registered [ 6078.492290] Key type .llcrypt registered [ 6078.527402] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 6078.875874] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6079.026447] Lustre: Lustre: Build Version: 2.15.7_5_g4a1d30d [ 6079.096808] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 6079.098553] LNet: Accept secure, port 988 [ 6080.680139] Key type lgssc registered [ 6081.193265] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6085.082385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6087.953515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6090.650814] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6093.847028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6093.879481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6094.977790] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6094.987571] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 6095.023852] Lustre: lustre-MDT0000: new disk, initializing [ 6095.045588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6095.051614] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6097.081213] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 6101.324120] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6101.345565] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6101.373101] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6101.386765] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 6101.388297] Lustre: Skipped 1 previous similar message [ 6101.420742] Lustre: lustre-MDT0001: new disk, initializing [ 6101.446866] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6101.456168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6101.458951] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6103.296470] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 6107.135572] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6108.779472] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6110.787082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6110.808224] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6110.811855] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 6110.895226] Lustre: lustre-OST0000: new disk, initializing [ 6110.897028] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 6110.921821] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6112.140771] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 6113.267442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6113.269844] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6115.591451] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6118.370022] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6118.370556] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6118.372705] Lustre: Skipped 1 previous similar message [ 6118.374869] Lustre: Skipped 2 previous similar messages [ 6123.491392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6123.495116] Lustre: Skipped 2 previous similar messages [ 6123.935449] Lustre: server umount lustre-MDT0000 complete [ 6125.781599] LustreError: 477977:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760457872 with bad export cookie 8289403926070569606 [ 6125.787970] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6125.788368] LustreError: 477977:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6125.942459] Lustre: server umount lustre-MDT0001 complete [ 6133.728184] Lustre: 481234:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760457874/real 1760457874] req@000000001650288f x1845973825369536/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1760457880 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 6133.845509] Lustre: server umount lustre-OST0000 complete [ 6139.193524] Lustre: DEBUG MARKER: == conf-sanity test 73: failnode to update from mountdata properly ========================================================== 12:04:46 (1760457886) [ 6139.749206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6143.021571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6143.230192] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6143.252804] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6144.915815] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 6147.900768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6148.009739] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6149.295866] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 6151.591056] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6153.431433] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6155.628804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6155.652894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6155.713927] Lustre: Found index 0 for lustre-OST0000, updating log [ 6157.134684] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 6158.821401] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5 to 0x0:33 [ 6159.536348] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6162.914730] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6162.915040] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6162.921659] Lustre: Skipped 1 previous similar message [ 6164.975229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6164.977682] Lustre: Skipped 1 previous similar message [ 6166.870282] Lustre: server umount lustre-OST0000 complete [ 6168.544681] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6168.547115] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6170.080872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6170.083266] Lustre: Skipped 3 previous similar messages [ 6180.321644] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6180.324857] Lustre: Skipped 3 previous similar messages [ 6182.368107] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 6182.410157] Lustre: server umount lustre-MDT0000 complete [ 6184.303711] LustreError: 484921:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760457931 with bad export cookie 8289403926070570929 [ 6184.309800] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6184.309870] LustreError: 484921:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6188.038210] Lustre: DEBUG MARKER: == conf-sanity test 75: The order of --index should be irrelevant ========================================================== 12:05:35 (1760457935) [ 6190.083522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6191.996716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6194.313978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6196.132569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6202.884645] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_hostid [ 6206.283273] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 6210.076578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6213.150614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6216.051562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6218.947240] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6221.587369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6221.605265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6221.675600] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6221.683673] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 6221.716724] Lustre: lustre-MDT0000: new disk, initializing [ 6221.738901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6221.740530] Lustre: Skipped 1 previous similar message [ 6221.745306] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6223.337666] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 6227.945637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6227.962686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6227.989029] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6228.002869] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 6228.004312] Lustre: Skipped 1 previous similar message [ 6228.037535] Lustre: lustre-MDT0001: new disk, initializing [ 6228.071742] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6228.074456] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6229.573896] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 6234.399422] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6236.985181] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6239.932961] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6239.969193] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6239.972856] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 6240.091945] Lustre: lustre-OST0000: new disk, initializing [ 6240.094443] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 6240.127507] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6240.129743] Lustre: Skipped 1 previous similar message [ 6241.974141] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all 8 [ 6246.819126] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6248.600293] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 6248.964489] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6248.970157] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6249.758123] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 6251.752829] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 6251.855068] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6254.051700] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6254.051739] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6254.054845] LustreError: Skipped 1 previous similar message [ 6254.058855] Lustre: Skipped 1 previous similar message [ 6254.059515] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6254.063075] Lustre: Skipped 3 previous similar messages [ 6258.872835] Lustre: server umount lustre-OST0000 complete [ 6258.874698] Lustre: Skipped 1 previous similar message [ 6263.776904] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6263.780655] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6263.786311] Lustre: Skipped 1 previous similar message [ 6267.880810] LustreError: 489067:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760458015 with bad export cookie 8289403926070571972 [ 6267.884655] LustreError: 166-1: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6267.889855] LustreError: 489067:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6278.312189] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_hostid [ 6281.634963] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 6286.452892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6289.562491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6292.386152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6295.449327] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6296.079808] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 6126 sec ========== 12:07:23 (1760458043) [ 6315.442152] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 6316.769118] Key type lgssc unregistered [ 6316.913534] LNet: 497805:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6316.918709] LNet: Removed LNI 192.168.203.156@tcp [ 6317.370974] Key type .llcrypt unregistered [ 6317.373717] Key type ._llcrypt unregistered