[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 506883255 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002315] x2apic enabled [ 0.003000] Switched APIC routing to physical x2apic. [ 0.003012] kvm-guest: setup PV IPIs [ 0.004000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.004000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.004016] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.005007] pid_max: default: 32768 minimum: 301 [ 0.006133] LSM: Security Framework initializing [ 0.007064] Yama: becoming mindful. [ 0.008041] SELinux: Initializing. [ 0.009060] *** VALIDATE selinux *** [ 0.021096] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026746] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027206] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028098] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029135] *** VALIDATE tmpfs *** [ 0.031359] *** VALIDATE proc *** [ 0.033051] *** VALIDATE cgroup *** [ 0.034000] *** VALIDATE cgroup2 *** [ 0.034292] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035165] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037054] Spectre V2 : User space: Vulnerable [ 0.038010] Speculative Store Bypass: Vulnerable [ 0.041927] debug: unmapping init [mem 0xffffffffb7859000-0xffffffffb7860fff] [ 0.044224] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046173] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047028] ... version: 2 [ 0.048008] ... bit width: 48 [ 0.049023] ... generic registers: 4 [ 0.050026] ... value mask: 0000ffffffffffff [ 0.051014] ... max period: 00007fffffffffff [ 0.052038] ... fixed-purpose events: 3 [ 0.053012] ... event mask: 000000070000000f [ 0.054550] rcu: Hierarchical SRCU implementation. [ 0.057821] smp: Bringing up secondary CPUs ... [ 0.059153] x86: Booting SMP configuration: [ 0.060011] .... node #0, CPUs: #1 #2 #3 [ 0.070124] smp: Brought up 1 node, 4 CPUs [ 0.072019] smpboot: Max logical packages: 1 [ 0.073018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.106000] node 0 deferred pages initialised in 30ms [ 0.110263] devtmpfs: initialized [ 0.111218] x86/mm: Memory block size: 128MB [ 0.114000] gcov: version magic: 0x41383552 [ 0.117491] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.123171] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.126987] pinctrl core: initialized pinctrl subsystem [ 0.128768] [ 0.129009] ************************************************************* [ 0.132022] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.135017] ** ** [ 0.137013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.139017] ** ** [ 0.141068] ** This means that this kernel is built to expose internal ** [ 0.144021] ** IOMMU data structures, which may compromise security on ** [ 0.146012] ** your system. ** [ 0.147000] ** ** [ 0.149015] ** If you see this message and you are not debugging the ** [ 0.151018] ** kernel, report this immediately to your vendor! ** [ 0.153016] ** ** [ 0.155017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.158016] ************************************************************* [ 0.162910] NET: Registered protocol family 16 [ 0.164615] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.167067] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.169139] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.173141] cpuidle: using governor menu [ 0.175144] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.178669] PCI: Using configuration type 1 for base access [ 0.181162] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.193342] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.194000] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.198654] cryptd: max_cpu_qlen set to 1000 [ 0.201628] ACPI: Added _OSI(Module Device) [ 0.202000] ACPI: Added _OSI(Processor Device) [ 0.202000] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.202000] ACPI: Added _OSI(Processor Aggregator Device) [ 0.208106] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.220920] ACPI: Interpreter enabled [ 0.222118] ACPI: PM: (supports S0 S3 S4 S5) [ 0.224020] ACPI: Using IOAPIC for interrupt routing [ 0.226427] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.230788] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.241392] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.243058] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.246030] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.249155] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.254836] acpiphp: Slot [2] registered [ 0.257233] acpiphp: Slot [3] registered [ 0.258111] acpiphp: Slot [4] registered [ 0.260122] acpiphp: Slot [5] registered [ 0.262167] acpiphp: Slot [6] registered [ 0.263136] acpiphp: Slot [7] registered [ 0.265146] acpiphp: Slot [8] registered [ 0.266189] acpiphp: Slot [9] registered [ 0.268269] acpiphp: Slot [10] registered [ 0.270231] acpiphp: Slot [11] registered [ 0.271105] acpiphp: Slot [12] registered [ 0.273166] acpiphp: Slot [13] registered [ 0.274131] acpiphp: Slot [14] registered [ 0.276140] acpiphp: Slot [15] registered [ 0.277295] acpiphp: Slot [16] registered [ 0.279124] acpiphp: Slot [17] registered [ 0.280259] acpiphp: Slot [18] registered [ 0.282126] acpiphp: Slot [19] registered [ 0.284140] acpiphp: Slot [20] registered [ 0.285095] acpiphp: Slot [21] registered [ 0.287145] acpiphp: Slot [22] registered [ 0.288124] acpiphp: Slot [23] registered [ 0.290139] acpiphp: Slot [24] registered [ 0.291101] acpiphp: Slot [25] registered [ 0.292139] acpiphp: Slot [26] registered [ 0.293113] acpiphp: Slot [27] registered [ 0.295560] acpiphp: Slot [28] registered [ 0.297126] acpiphp: Slot [29] registered [ 0.298512] acpiphp: Slot [30] registered [ 0.300154] acpiphp: Slot [31] registered [ 0.301000] PCI host bridge to bus 0000:00 [ 0.301000] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.303037] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.305030] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.308037] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.311040] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.317035] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.320242] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.325066] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.327958] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.338789] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.344069] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.347039] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.349029] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.352031] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.355789] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.360114] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.363061] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.365843] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.373029] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.390896] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.394024] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.400910] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.415023] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.424026] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.456018] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.471495] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.483020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.494022] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.519018] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.532252] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.542016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.552029] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.566015] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.575489] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.588014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.594021] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.612019] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.622443] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.628019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.634021] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.653031] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.668169] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.680023] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.694053] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.711042] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.723045] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.725690] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.728399] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.731485] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.733197] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.738055] iommu: Default domain type: Passthrough [ 0.739045] SCSI subsystem initialized [ 0.740213] ACPI: bus type USB registered [ 0.742143] usbcore: registered new interface driver usbfs [ 0.745482] usbcore: registered new interface driver hub [ 0.746137] usbcore: registered new device driver usb [ 0.749228] pps_core: LinuxPPS API ver. 1 registered [ 0.751011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.755492] PTP clock support registered [ 0.759177] EDAC MC: Ver: 3.0.0 [ 0.763228] PCI: Using ACPI for IRQ routing [ 0.765802] NetLabel: Initializing [ 0.767022] NetLabel: domain hash size = 128 [ 0.769031] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.771178] NetLabel: unlabeled traffic allowed by default [ 0.773264] vgaarb: loaded [ 0.775376] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.778194] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.784930] clocksource: Switched to clocksource kvm-clock [ 0.932174] VFS: Disk quotas dquot_6.6.0 [ 0.933973] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.937140] *** VALIDATE ramfs *** [ 0.938633] *** VALIDATE hugetlbfs *** [ 0.941196] pnp: PnP ACPI init [ 0.944780] pnp: PnP ACPI: found 6 devices [ 0.975508] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.980959] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.983383] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.987015] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.990588] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.993496] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.996415] NET: Registered protocol family 2 [ 0.999374] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.006307] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.010446] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.015971] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.019424] TCP: Hash tables configured (established 65536 bind 65536) [ 1.022436] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.025713] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.028902] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.033567] NET: Registered protocol family 1 [ 1.052880] RPC: Registered named UNIX socket transport module. [ 1.054459] RPC: Registered udp transport module. [ 1.058118] RPC: Registered tcp transport module. [ 1.060334] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.065637] NET: Registered protocol family 44 [ 1.071154] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.074777] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.078824] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.086701] PCI: CLS 0 bytes, default 64 [ 1.096326] Unpacking initramfs... [ 2.759280] debug: unmapping init [mem 0xffff8df3fcc54000-0xffff8df3fffbffff] [ 2.763644] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.765525] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.767812] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.379698] Initialise system trusted keyrings [ 3.381417] Key type blacklist registered [ 3.384180] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.407730] zbud: loaded [ 3.421595] *** VALIDATE nfs *** [ 3.422931] *** VALIDATE nfs4 *** [ 3.424798] pstore: using deflate compression [ 3.436573] Platform Keyring initialized [ 3.751155] NET: Registered protocol family 38 [ 3.752742] Key type asymmetric registered [ 3.753971] Asymmetric key parser 'x509' registered [ 3.755460] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.759691] io scheduler mq-deadline registered [ 3.761344] io scheduler kyber registered [ 3.762843] io scheduler bfq registered [ 3.764575] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.775471] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.778177] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.781544] ACPI: Power Button [PWRF] [ 3.882343] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.979504] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.203485] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.417040] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.658376] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.702269] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.735095] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.743926] Non-volatile memory driver v1.3 [ 4.745397] Linux agpgart interface v0.103 [ 4.812754] virtio_blk virtio1: [vda] 67968 512-byte logical blocks (34.8 MB/33.2 MiB) [ 4.817297] vda: detected capacity change from 0 to 34799616 [ 4.842867] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.846055] vdb: detected capacity change from 0 to 1073741824 [ 4.883264] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.889590] vdc: detected capacity change from 0 to 2621440000 [ 4.926903] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.932163] vdd: detected capacity change from 0 to 2621440000 [ 4.965859] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.974778] vde: detected capacity change from 0 to 4294967296 [ 5.019063] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.029154] vdf: detected capacity change from 0 to 4294967296 [ 5.047737] libphy: Fixed MDIO Bus: probed [ 5.069363] usbcore: registered new interface driver usbserial_generic [ 5.074112] usbserial: USB Serial support registered for generic [ 5.077453] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.084739] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.086724] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.091676] mousedev: PS/2 mouse device common for all mice [ 5.107384] rtc_cmos 00:05: RTC can wake from S4 [ 5.115466] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.123594] rtc_cmos 00:05: registered as rtc0 [ 5.123703] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.123761] intel_pstate: CPU model not supported [ 5.130457] hid: raw HID events driver (C) Jiri Kosina [ 5.145597] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.151727] usbcore: registered new interface driver usbhid [ 5.151738] usbhid: USB HID core driver [ 5.151892] drop_monitor: Initializing network drop monitor service [ 5.152055] Initializing XFRM netlink socket [ 5.152383] NET: Registered protocol family 10 [ 5.170120] Segment Routing with IPv6 [ 5.170165] NET: Registered protocol family 17 [ 5.177402] mpls_gso: MPLS GSO support [ 5.186188] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.205344] RAS: Correctable Errors collector initialized. [ 5.207875] AVX version of gcm_enc/dec engaged. [ 5.212675] AES CTR mode by8 optimization enabled [ 5.397658] sched_clock: Marking stable (5397632091, 0)->(6561197274, -1163565183) [ 5.405849] registered taskstats version 1 [ 5.408748] Loading compiled-in X.509 certificates [ 5.411875] zswap: loaded using pool lzo/zbud [ 5.474870] Key type big_key registered [ 5.503664] Key type encrypted registered [ 5.505069] ima: No TPM chip found, activating TPM-bypass! [ 5.507060] ima: Allocated hash algorithm: sha1 [ 5.508809] ima: No architecture policies found [ 5.510445] evm: Initialising EVM extended attributes: [ 5.511825] evm: security.selinux [ 5.512633] evm: security.ima [ 5.513244] evm: security.capability [ 5.514070] evm: HMAC attrs: 0x1 [ 5.516916] rtc_cmos 00:05: setting system clock to 2025-10-24 12:52:36 UTC (1761310356) [ 5.546247] debug: unmapping init [mem 0xffffffffb8803000-0xffffffffb89fffff] [ 5.549847] debug: unmapping init [mem 0xffffffffb7582000-0xffffffffb7858fff] [ 5.562131] Write protecting the kernel read-only data: 28672k [ 5.567161] debug: unmapping init [mem 0xffffffffb5c03000-0xffffffffb5dfffff] [ 5.571331] debug: unmapping init [mem 0xffffffffb6514000-0xffffffffb65fffff] [ 5.636975] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.646410] systemd[1]: Detected virtualization kvm. [ 5.648559] systemd[1]: Detected architecture x86-64. [ 5.650213] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.679577] systemd[1]: No hostname configured. [ 5.682384] systemd[1]: Set hostname to . [ 5.684313] random: systemd: uninitialized urandom read (16 bytes read) [ 5.689231] systemd[1]: Initializing machine ID from random generator. [ 5.812726] random: ln: uninitialized urandom read (6 bytes read) [ 6.028374] random: systemd: uninitialized urandom read (16 bytes read) [ 6.031506] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 6.046620] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 6.054980] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Reached target Local File Systems. [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Swap. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.491694] device-mapper: uevent: version 1.0.3 [ 7.496812] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ 9.122038] virtio_net virtio0 ens2: renamed from eth0 Starting dracut initqueue hook... [[ 9.169483] random: fast init done  OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 9.306926] scsi host0: ata_piix [ 9.449276] scsi host1: ata_piix [ 9.456839] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 9.472551] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 15.032900] random: crng init done [ 15.038224] random: 7 urandom warning(s) missed due to ratelimiting [ 17.951227] dracut-initqueue[593]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 19.064694] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 21.262518] printk: systemd: 26 output lines suppressed due to ratelimiting [ 21.729765] SELinux: Disabled at runtime. [ 21.819106] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 21.831900] systemd[1]: Detected virtualization kvm. [ 21.833831] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 23.179393] systemd[1]: initrd-switch-root.service: Succeeded. [ 23.184477] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 23.205686] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 23.215475] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 23.219866] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 23.238392] systemd[1]: Starting Journal Service... Starting Journal Service... [ 23.253668] systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... Mounting Kernel Debug File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Forward Password Requests to Wall Directory Watch. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on Process Core Dump Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Kernel Socket. Mounting POSIX Message Queue File System... [ 23.725763] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... [ OK ] Listening on initctl Compatibility Named Pipe. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Reached target Local Encrypted Volumes. Mounting Huge Pages File System... [ OK ] Created slice User and Session Slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target Slices. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Paths. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 25.046959] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 26.272258] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 26.395821] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 27.506848] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 27.777116] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit)[ 32.341936] Key type dns_resolver registered [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 32.844899] NFS: Registering the id_resolver key type [ 32.846634] Key type id_resolver registered [ 32.848655] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ ***] A start job is running for Configur…only root support (10s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Login Service. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg256-server login: [ 75.995893] libcfs: loading out-of-tree module taints kernel. [ 76.022978] alg: No test for adler32 (adler32-zlib) [ 76.775343] Key type ._llcrypt registered [ 76.777952] Key type .llcrypt registered [ 76.837642] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 87.044654] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 87.674126] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 87.986635] Lustre: Lustre: Build Version: 2.15.7_9_g1c128dc [ 88.355225] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 88.358523] LNet: Accept secure, port 988 [ 90.024189] Key type lgssc registered [ 90.809804] Lustre: Echo OBD driver; http://www.lustre.org/ [ 100.296062] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 101.655102] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 107.053304] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 111.186232] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 115.293605] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 123.157775] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 129.401482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 129.453024] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 130.574745] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 130.589981] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 130.650044] Lustre: lustre-MDT0000: new disk, initializing [ 130.725803] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 130.743995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 133.066910] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 141.016408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 141.071618] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 141.106531] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 141.127448] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 141.130613] Lustre: Skipped 1 previous similar message [ 141.181967] Lustre: lustre-MDT0001: new disk, initializing [ 141.216523] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 141.233433] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 141.237680] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 143.318799] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 150.184640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 150.219745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 150.223725] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 150.343122] Lustre: lustre-OST0000: new disk, initializing [ 150.346826] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 150.379110] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 152.243630] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 156.673917] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 156.682619] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 159.787766] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 159.851620] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 159.856603] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 159.934833] Lustre: lustre-OST0001: new disk, initializing [ 159.937741] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 159.979662] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 162.128526] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 163.076519] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 163.080455] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 169.032735] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 175.509274] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 182.926874] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing check_logdir /tmp/testlogs/ [ 185.328350] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing yml_node [ 187.600258] Lustre: DEBUG MARKER: Client: 2.15.7.9 [ 188.871064] Lustre: DEBUG MARKER: MDS: 2.15.7.9 [ 190.185562] Lustre: DEBUG MARKER: OSS: 2.15.7.9 [ 191.138328] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Oct 24 08:55:41 EDT 2025 [ 195.626252] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball 110 [ 196.399545] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 [ 209.378854] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 209.379786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 209.389284] Lustre: Skipped 1 previous similar message [ 209.398399] Lustre: Skipped 3 previous similar messages [ 214.496846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 214.499589] Lustre: Skipped 3 previous similar messages [ 215.142758] Lustre: server umount lustre-MDT0000 complete [ 217.566174] LustreError: 6225:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761310568 with bad export cookie 6460024546366471429 [ 217.567585] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 217.571782] LustreError: 6225:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 219.617697] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 219.620810] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 219.621814] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 219.625739] Lustre: Skipped 3 previous similar messages [ 219.633131] LustreError: Skipped 1 previous similar message [ 222.625382] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 222.626431] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 222.645416] Lustre: Skipped 2 previous similar messages [ 223.650746] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 223.667739] LustreError: Skipped 2 previous similar messages [ 223.830813] Lustre: server umount lustre-MDT0001 complete [ 226.274259] Lustre: server umount lustre-OST0000 complete [ 228.474267] Lustre: server umount lustre-OST0001 complete [ 232.352280] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 237.284443] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 242.971329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 246.840874] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 250.394107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 254.072381] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 258.134799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 258.196573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 258.347417] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 258.366814] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 258.410772] Lustre: lustre-MDT0000: new disk, initializing [ 258.460673] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 258.470266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 261.000526] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 268.300684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 268.357119] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 268.418334] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 268.449792] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 268.453216] Lustre: Skipped 1 previous similar message [ 268.508599] Lustre: lustre-MDT0001: new disk, initializing [ 268.584755] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 268.618900] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 268.626633] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 271.345727] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 277.852142] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 281.508211] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 285.925407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 285.991417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 285.995760] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 286.223548] Lustre: lustre-OST0000: new disk, initializing [ 286.228212] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 286.300236] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 287.728960] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 287.735150] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 289.156367] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 294.836430] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 297.025563] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 297.134939] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 299.276331] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 299.378518] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 303.074464] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 303.075593] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 303.076242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 303.076246] Lustre: Skipped 2 previous similar messages [ 303.079073] Lustre: Skipped 1 previous similar message [ 306.417148] Lustre: server umount lustre-OST0000 complete [ 309.729194] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 309.733185] LustreError: Skipped 1 previous similar message [ 309.738893] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 313.315637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 313.321257] Lustre: Skipped 3 previous similar messages [ 314.838639] Lustre: server umount lustre-MDT0000 complete [ 316.810434] LustreError: 14301:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761310667 with bad export cookie 6460024546366472885 [ 316.814651] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 316.814922] LustreError: 14301:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 319.073524] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 08:57:49 (1761310669) [ 320.434600] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 321.419876] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 08:57:51 (1761310671) [ 324.867909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 325.113547] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 325.152661] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 327.295556] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 330.211082] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 331.543165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 333.883524] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 337.987695] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 341.108193] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 344.777854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 346.922557] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 350.916730] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 360.421676] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 360.421857] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 360.427586] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 360.433839] Lustre: Skipped 2 previous similar messages [ 360.438897] Lustre: Skipped 1 previous similar message [ 366.188384] Lustre: server umount lustre-OST0000 complete [ 366.190845] Lustre: Skipped 1 previous similar message [ 370.659201] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 370.670388] Lustre: Skipped 1 previous similar message [ 375.779476] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 375.788042] LustreError: Skipped 1 previous similar message [ 376.083209] LustreError: 18835:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761310727 with bad export cookie 6460024546366473739 [ 376.086835] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 376.090486] LustreError: 18835:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 382.602052] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 383.983446] Key type lgssc unregistered [ 384.132481] LNet: 22509:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 384.137952] LNet: Removed LNI 192.168.202.156@tcp [ 384.560794] Key type .llcrypt unregistered [ 384.562525] Key type ._llcrypt unregistered [ 397.037241] alg: No test for adler32 (adler32-zlib) [ 397.789532] Key type ._llcrypt registered [ 397.791301] Key type .llcrypt registered [ 397.847970] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 398.414272] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 398.604944] Lustre: Lustre: Build Version: 2.15.7_9_g1c128dc [ 398.720736] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 398.724638] LNet: Accept secure, port 988 [ 400.360228] Key type lgssc registered [ 400.866561] Lustre: Echo OBD driver; http://www.lustre.org/ [ 404.889553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 406.094136] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 406.120774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 408.095323] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 411.617996] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 412.507481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 412.715660] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 414.992780] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 419.482826] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 422.597322] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 425.963866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 426.134845] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 428.220332] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 432.121922] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 436.391414] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 446.948793] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 446.948835] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 446.949864] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 446.957978] Lustre: Skipped 1 previous similar message [ 449.326434] Lustre: server umount lustre-OST0000 complete [ 453.607808] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 453.618631] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 453.648289] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 453.650668] Lustre: Skipped 1 previous similar message [ 457.187480] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 457.189805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 457.261991] Lustre: Skipped 1 previous similar message [ 459.406362] Lustre: server umount lustre-MDT0000 complete [ 462.305291] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 462.335354] LustreError: Skipped 1 previous similar message [ 464.370469] LustreError: 23952:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761310815 with bad export cookie 6693414047474996606 [ 464.378520] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 464.405777] LustreError: 23952:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 464.582134] hrtimer: interrupt took 4365292 ns [ 465.116891] Lustre: server umount lustre-MDT0001 complete [ 482.697573] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 486.194739] Key type lgssc unregistered [ 486.497545] LNet: 27647:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 486.511184] LNet: Removed LNI 192.168.202.156@tcp [ 487.680712] Key type .llcrypt unregistered [ 487.687115] Key type ._llcrypt unregistered [ 509.321562] alg: No test for adler32 (adler32-zlib) [ 510.077515] Key type ._llcrypt registered [ 510.083487] Key type .llcrypt registered [ 510.256963] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 511.372652] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 512.051986] Lustre: Lustre: Build Version: 2.15.7_9_g1c128dc [ 512.270125] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 512.279607] LNet: Accept secure, port 988 [ 513.936164] Key type lgssc registered [ 515.015423] Lustre: Echo OBD driver; http://www.lustre.org/ [ 521.135948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 522.479937] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 522.531565] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 525.769432] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 527.841449] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 532.961379] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 533.056240] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 533.306110] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 536.316125] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 542.242365] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 546.466031] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 551.791547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 552.092670] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 553.161202] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 555.320767] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 561.406689] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 573.058529] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.56@tcp (stopping) [ 573.924851] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 573.927174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 573.938796] Lustre: Skipped 2 previous similar messages [ 573.944051] Lustre: Skipped 2 previous similar messages [ 575.738756] Lustre: server umount lustre-MDT0000 complete [ 578.158226] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 578.555405] LustreError: 29102:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761310929 with bad export cookie 13416127223310228244 [ 578.557770] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 578.565396] LustreError: 29102:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 579.042786] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 579.049953] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 583.073455] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 583.073521] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 583.081721] LustreError: Skipped 1 previous similar message [ 585.004264] Lustre: server umount lustre-MDT0001 complete [ 604.937710] Lustre: server umount lustre-OST0000 complete [ 617.637806] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 619.761987] Key type lgssc unregistered [ 619.994077] LNet: 33108:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 620.004128] LNet: Removed LNI 192.168.202.156@tcp [ 620.560156] Key type .llcrypt unregistered [ 620.562818] Key type ._llcrypt unregistered [ 636.426174] alg: No test for adler32 (adler32-zlib) [ 637.177336] Key type ._llcrypt registered [ 637.180561] Key type .llcrypt registered [ 637.259752] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 637.909581] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 638.138645] Lustre: Lustre: Build Version: 2.15.7_9_g1c128dc [ 638.272114] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 638.274761] LNet: Accept secure, port 988 [ 639.920367] Key type lgssc registered [ 640.648502] Lustre: Echo OBD driver; http://www.lustre.org/ [ 646.356833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 647.660202] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 647.702453] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 650.207907] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 652.771456] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 656.007666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 656.252594] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 659.607614] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 665.286862] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 669.721885] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 674.535258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 674.770721] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 677.841948] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 677.882128] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 683.088159] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 693.222643] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 693.230541] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 693.255817] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 694.918417] Lustre: lustre-OST0000: Not available for connect from 192.168.202.56@tcp (stopping) [ 696.292267] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 696.295246] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 696.295774] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 697.585378] Lustre: server umount lustre-OST0000 complete [ 713.303184] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 713.303878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 713.318232] Lustre: Skipped 1 previous similar message [ 713.324098] Lustre: Skipped 1 previous similar message [ 717.347181] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 717.352764] Lustre: Skipped 1 previous similar message [ 719.487899] Lustre: server umount lustre-MDT0000 complete [ 722.213981] LustreError: 34550:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761311073 with bad export cookie 1938403876473767494 [ 722.222395] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 722.226889] LustreError: 34550:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 728.527845] Lustre: server umount lustre-MDT0001 complete [ 735.636603] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 737.505310] Key type lgssc unregistered [ 737.700319] LNet: 38397:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 737.706983] LNet: Removed LNI 192.168.202.156@tcp [ 738.265376] Key type .llcrypt unregistered [ 738.268455] Key type ._llcrypt unregistered [ 752.407428] alg: No test for adler32 (adler32-zlib) [ 753.159477] Key type ._llcrypt registered [ 753.162094] Key type .llcrypt registered [ 753.226296] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 753.707155] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 754.050729] Lustre: Lustre: Build Version: 2.15.7_9_g1c128dc [ 754.215321] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 754.219568] LNet: Accept secure, port 988 [ 755.827263] Key type lgssc registered [ 756.432685] Lustre: Echo OBD driver; http://www.lustre.org/ [ 761.152778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 762.445696] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 762.500138] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 765.298539] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 767.974312] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 771.018779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 771.238539] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 773.805455] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 778.548167] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 782.158942] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 786.219142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 786.412293] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 788.656390] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 792.558922] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 792.970330] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 796.824121] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 798.820688] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 804.338903] LNetError: 120-3: Refusing connection from 192.168.202.156 for 127.0.0.2@tcp: No matching NI [ 804.345784] LNetError: 39174:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 804.350073] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 810.224602] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2025-10-24 9h06m00s [ 811.094761] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 812.513862] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 812.517583] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 812.527374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 813.027284] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 813.033788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 813.035035] LNetError: 120-3: Refusing connection from 192.168.202.156 for 127.0.0.2@tcp: No matching NI [ 813.036184] Lustre: Skipped 2 previous similar messages [ 813.040121] LNetError: 39171:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 813.049621] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 814.739125] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.56@tcp (stopping) [ 818.146825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 818.149663] Lustre: Skipped 2 previous similar messages [ 818.165063] LNetError: 120-3: Refusing connection from 192.168.202.156 for 127.0.0.2@tcp: No matching NI [ 818.171498] LNetError: 39172:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 818.178423] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 818.398611] Lustre: server umount lustre-MDT0000 complete [ 819.824357] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 820.334527] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 823.271327] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 823.277202] LustreError: Skipped 2 previous similar messages [ 823.284900] LNetError: 120-3: Refusing connection from 192.168.202.156 for 127.0.0.2@tcp: No matching NI [ 823.288101] LNetError: 39173:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 823.292031] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 824.446660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 824.526731] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 824.536820] Lustre: Evicted from MGS (at 192.168.202.156@tcp) after server handle changed from 0xe5314ad0af881025 to 0xe5314ad0af881477 [ 824.544553] Lustre: MGC192.168.202.156@tcp: Connection restored to (at 0@lo) [ 824.679867] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 827.318986] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 829.491190] Lustre: DEBUG MARKER: Wait for df (28864) ... [ 829.926232] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 829.937721] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to (at 0@lo) [ 829.940613] Lustre: Skipped 1 previous similar message [ 830.457163] Lustre: DEBUG MARKER: done [ 833.507061] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 833.510704] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 833.516885] Lustre: Skipped 1 previous similar message [ 833.520161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 845.284217] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 845.292056] Lustre: Skipped 5 previous similar messages [ 847.328693] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 847.443933] Lustre: server umount lustre-OST0000 complete [ 850.401041] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 850.407201] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 850.416374] Lustre: Skipped 2 previous similar messages [ 850.422767] LNetError: 120-3: Refusing connection from 192.168.202.156 for 127.0.0.2@tcp: No matching NI [ 850.427913] LNetError: 39174:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 850.430916] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 864.224385] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 864.314672] Lustre: server umount lustre-MDT0000 complete [ 865.765156] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 865.773247] LustreError: Skipped 4 previous similar messages [ 866.502279] LustreError: 42569:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761311217 with bad export cookie 16515063568702903415 [ 866.505476] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 866.507321] LustreError: 42569:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 866.672963] Lustre: server umount lustre-MDT0001 complete [ 873.179623] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 874.753954] Key type lgssc unregistered [ 874.937423] LNet: 44968:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 874.944666] LNet: Removed LNI 192.168.202.156@tcp [ 875.430466] Key type .llcrypt unregistered [ 875.432511] Key type ._llcrypt unregistered [ 894.120925] alg: No test for adler32 (adler32-zlib) [ 894.875686] Key type ._llcrypt registered [ 894.878869] Key type .llcrypt registered [ 894.948219] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 903.075321] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 903.613963] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 903.789681] Lustre: Lustre: Build Version: 2.15.7_9_g1c128dc [ 903.895166] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 903.898148] LNet: Accept secure, port 988 [ 905.496666] Key type lgssc registered [ 906.024705] Lustre: Echo OBD driver; http://www.lustre.org/ [ 910.445704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 913.513112] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 916.546826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 919.607232] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 923.555317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 923.589077] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 924.757449] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 924.780314] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 924.831080] Lustre: lustre-MDT0000: new disk, initializing [ 924.891384] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 924.904284] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 927.301750] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 933.494992] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 933.545507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 933.613866] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 933.648791] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 933.651629] Lustre: Skipped 1 previous similar message [ 933.693619] Lustre: lustre-MDT0001: new disk, initializing [ 933.737629] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 933.760793] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 933.764576] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 936.643626] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 948.144846] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 955.371351] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 963.581573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 963.644892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 963.648187] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 963.832599] Lustre: lustre-OST0000: new disk, initializing [ 963.838560] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 963.958865] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 968.224917] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 973.900155] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 973.911976] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 976.677151] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 980.572172] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 980.711108] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 983.987845] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 984.169241] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 989.667476] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 989.671307] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 989.675985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 989.677629] LustreError: Skipped 1 previous similar message [ 989.683582] Lustre: Skipped 1 previous similar message [ 991.686052] Lustre: server umount lustre-OST0000 complete [ 994.787376] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 994.793321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 994.802318] Lustre: Skipped 2 previous similar messages [ 994.810084] Lustre: Skipped 1 previous similar message [ 999.907242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 999.917155] Lustre: Skipped 1 previous similar message [ 1000.586371] Lustre: server umount lustre-MDT0000 complete [ 1002.976605] LustreError: 49922:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761311353 with bad export cookie 2217172846690939093 [ 1002.981156] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1002.983209] LustreError: 49922:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1003.167248] Lustre: server umount lustre-MDT0001 complete [ 1008.707462] Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 09:09:19 (1761311359) [ 1013.526731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1013.838327] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1013.883726] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1016.414750] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 1019.362780] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1021.188676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1021.404296] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1023.872547] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 1028.520651] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1032.211301] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1036.307237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1036.509331] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1039.099287] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 1044.830630] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1050.576997] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2025-10-24 9h10m00s [ 1052.317385] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1062.243055] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1062.380709] LNetError: 120-3: Refusing connection from 192.168.202.156 for 127.0.0.2@tcp: No matching NI [ 1062.384621] LNetError: 47277:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1062.417770] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1065.539535] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1065.594379] LustreError: 53610:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1086.104640] Lustre: lustre-MDT0000: Client 61d3a5c3-b9ee-4456-8255-58e663831309 (at 192.168.202.56@tcp) reconnecting [ 1105.664161] LustreError: 53610:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 136 awake [ 1106.919564] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1108.777850] Lustre: DEBUG MARKER: done [ 1114.593475] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1114.601563] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1114.611697] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1116.640956] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1116.661789] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1120.603402] Lustre: server umount lustre-OST0000 complete [ 1126.885743] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1126.889572] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1126.900869] Lustre: Skipped 1 previous similar message [ 1126.905116] LNetError: 120-3: Refusing connection from 192.168.202.156 for 127.0.0.2@tcp: No matching NI [ 1126.909504] Lustre: Skipped 3 previous similar messages [ 1126.911074] LNetError: 47274:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1126.932271] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1130.570668] Lustre: server umount lustre-MDT0000 complete [ 1132.002955] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1132.013050] LNetError: 120-3: Refusing connection from 192.168.202.156 for 127.0.0.2@tcp: No matching NI [ 1132.027616] LNetError: 47275:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1132.028883] LustreError: Skipped 1 previous similar message [ 1132.031965] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1134.200757] LustreError: 53595:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761311485 with bad export cookie 2217172846690939954 [ 1134.211535] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1134.235366] LustreError: 53595:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1145.368417] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 1147.655542] Key type lgssc unregistered [ 1147.879793] LNet: 58026:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1147.886418] LNet: Removed LNI 192.168.202.156@tcp [ 1148.550095] Key type .llcrypt unregistered [ 1148.551953] Key type ._llcrypt unregistered [ 1176.101953] alg: No test for adler32 (adler32-zlib) [ 1176.864467] Key type ._llcrypt registered [ 1176.866476] Key type .llcrypt registered [ 1176.968197] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 1186.680690] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 1187.337969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1187.524350] Lustre: Lustre: Build Version: 2.15.7_9_g1c128dc [ 1187.692322] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 1187.701824] LNet: Accept secure, port 988 [ 1189.352153] Key type lgssc registered [ 1190.147758] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1195.679417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1200.107602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1204.581986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1209.046590] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1215.110874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1215.197270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1216.400334] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1216.438059] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1216.518238] Lustre: lustre-MDT0000: new disk, initializing [ 1216.617289] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1216.641911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1220.175813] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 1228.420860] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1228.515399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1228.615179] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1228.652976] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1228.660659] Lustre: Skipped 1 previous similar message [ 1228.707847] Lustre: lustre-MDT0001: new disk, initializing [ 1228.783612] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1228.810046] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1228.830641] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1232.878207] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 1241.167488] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1245.769283] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1251.617662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1251.677437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1251.682403] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1251.866163] Lustre: lustre-OST0000: new disk, initializing [ 1251.869979] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1251.935287] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1255.144328] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 1258.032470] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1258.043722] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1262.234084] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1265.965765] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1266.123418] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1269.730088] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 1269.970466] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1273.829408] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1273.832521] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1273.844085] LustreError: Skipped 1 previous similar message [ 1273.846347] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1273.851575] Lustre: Skipped 1 previous similar message [ 1277.597259] Lustre: server umount lustre-OST0000 complete [ 1284.065718] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1284.083759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1284.086713] Lustre: Skipped 1 previous similar message [ 1287.354602] Lustre: server umount lustre-MDT0000 complete [ 1289.185784] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1289.200299] LustreError: Skipped 1 previous similar message [ 1290.307542] LustreError: 62114:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761311641 with bad export cookie 10278546387391206888 [ 1290.312649] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1290.557168] Lustre: server umount lustre-MDT0001 complete [ 1297.557483] Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 09:14:07 (1761311647) [ 1298.960942] Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST [ 1300.349551] Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 09:14:10 (1761311650) [ 1310.667621] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 1315.422662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1316.157443] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1316.274868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1317.626089] Lustre: Failing over lustre-MDT0000 [ 1317.637261] LustreError: 67137:0:(osp_object.c:629:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1317.652151] LustreError: 67137:0:(lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1317.665081] LustreError: 67137:0:(lod_dev.c:424:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000 get update log failed: rc = -5 [ 1324.000812] Lustre: 67188:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761311668/real 1761311668] req@00000000580318f0 x1846869007290176/t0(0) o251->MGC192.168.202.156@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761311674 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1324.191705] Lustre: server umount lustre-MDT0000 complete [ 1329.769244] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 09:14:39 (1761311679) [ 1335.054069] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1335.378177] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1335.443491] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1339.168521] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 1340.917107] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1346.021815] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1346.305427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1346.562845] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1350.043409] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 1356.873817] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1362.749333] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1369.215791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1369.434660] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1372.667752] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 1378.440204] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1392.893118] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1400.500747] Lustre: server umount lustre-MDT0000 complete [ 1401.316036] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1401.317254] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1401.327643] Lustre: Skipped 2 previous similar messages [ 1403.187752] LustreError: 67737:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761311754 with bad export cookie 10278546387391208015 [ 1403.188590] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1403.192477] LustreError: 67737:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1405.875574] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1409.611987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1413.472851] Lustre: 60341:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761311757/real 1761311757] req@000000005250a296 x1846869007308672/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761311764 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1413.501540] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1413.514177] Lustre: Skipped 1 previous similar message [ 1414.908277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1415.648253] Lustre: 60342:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761311759/real 1761311759] req@00000000b01046b5 x1846869007308736/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761311766 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1418.658978] Lustre: 60343:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761311762/real 1761311762] req@00000000a29ef009 x1846869007308928/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761311769 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1418.682898] Lustre: 60343:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1420.708263] Lustre: Evicted from MGS (at 192.168.202.156@tcp) after server handle changed from 0x8ea4bb7b33d3da4f to 0x8ea4bb7b33d3e1e9 [ 1420.718357] Lustre: MGC192.168.202.156@tcp: Connection restored to 192.168.202.156@tcp (at 0@lo) [ 1421.036920] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1421.054356] LustreError: Skipped 2 previous similar messages [ 1421.101560] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1425.108339] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 1431.374265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1431.707374] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:65 [ 1433.702303] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 1435.134125] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 1440.398447] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1444.691821] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1449.443389] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1449.446165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1449.458190] Lustre: Skipped 2 previous similar messages [ 1449.458352] Lustre: Skipped 1 previous similar message [ 1451.309299] Lustre: server umount lustre-MDT0000 complete [ 1451.311186] Lustre: Skipped 1 previous similar message [ 1453.593164] LustreError: 67736:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761311804 with bad export cookie 10278546387391209961 [ 1453.596591] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1453.599984] LustreError: 67736:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 1457.790515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1461.664232] Lustre: 60342:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761311805/real 1761311805] req@0000000031ee3bff x1846869007322112/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761311812 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1461.698789] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1464.462475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1470.947411] Lustre: Evicted from MGS (at 192.168.202.156@tcp) after server handle changed from 0x8ea4bb7b33d3e1e9 to 0x8ea4bb7b33d3e7f4 [ 1470.969403] Lustre: MGC192.168.202.156@tcp: Connection restored to (at 0@lo) [ 1470.972309] Lustre: Skipped 1 previous similar message [ 1471.201608] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1471.207581] LustreError: Skipped 5 previous similar messages [ 1471.341197] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 1474.588661] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 1481.710103] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 1482.169822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1482.555421] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:97 [ 1486.384154] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 1487.853265] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 1492.870109] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1499.216525] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1507.092633] Lustre: server umount lustre-MDT0000 complete [ 1507.095633] Lustre: Skipped 1 previous similar message [ 1508.322328] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1508.322430] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1508.341333] Lustre: Skipped 2 previous similar messages [ 1510.479332] LustreError: 67737:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761311861 with bad export cookie 10278546387391211508 [ 1510.482390] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1510.495141] LustreError: 67737:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1516.802144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1520.065051] Lustre: DEBUG MARKER: files compared the same [ 1520.480575] Lustre: 60343:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761311864/real 1761311864] req@0000000028234573 x1846869007334720/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761311871 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1520.508103] Lustre: 60343:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 1538.393739] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 1540.913575] Key type lgssc unregistered [ 1541.115714] LNet: 77823:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1541.126508] LNet: Removed LNI 192.168.202.156@tcp [ 1541.767910] Key type .llcrypt unregistered [ 1541.769460] Key type ._llcrypt unregistered [ 1559.502730] alg: No test for adler32 (adler32-zlib) [ 1560.267544] Key type ._llcrypt registered [ 1560.270539] Key type .llcrypt registered [ 1560.382450] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 1561.027522] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1561.425471] Lustre: Lustre: Build Version: 2.15.7_9_g1c128dc [ 1561.587421] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 1561.593147] LNet: Accept secure, port 988 [ 1563.224361] Key type lgssc registered [ 1564.172672] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1570.965791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1572.342017] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1572.390445] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1575.830610] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug +malloc all 8 [ 1577.443395] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1582.566823] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1582.710529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1583.017795] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1586.829090] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug +malloc all 8 [ 1593.559819] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1599.247487] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1604.501825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1604.736916] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1608.149792] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug +malloc all 8 [ 1612.906305] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:129 [ 1613.780446] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1622.127567] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 1632.745420] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1632.748286] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1632.756851] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1632.781656] Lustre: Skipped 1 previous similar message [ 1632.798107] Lustre: Skipped 1 previous similar message [ 1636.973138] Lustre: server umount lustre-OST0000 complete [ 1642.977454] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1642.978289] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1642.982316] Lustre: Skipped 1 previous similar message [ 1647.335704] Lustre: server umount lustre-MDT0000 complete [ 1648.099887] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1648.109384] LustreError: Skipped 1 previous similar message [ 1650.402517] LustreError: 82161:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761312001 with bad export cookie 13749753621145425919 [ 1650.409636] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1650.412081] LustreError: 82161:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1650.704902] Lustre: server umount lustre-MDT0001 complete [ 1660.039648] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 1662.288770] Key type lgssc unregistered [ 1662.487515] LNet: 82965:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1662.498462] LNet: Removed LNI 192.168.202.156@tcp [ 1663.094862] Key type .llcrypt unregistered [ 1663.097216] Key type ._llcrypt unregistered [ 1680.668914] alg: No test for adler32 (adler32-zlib) [ 1681.424432] Key type ._llcrypt registered [ 1681.426147] Key type .llcrypt registered [ 1681.557413] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 1682.485463] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1682.912383] Lustre: Lustre: Build Version: 2.15.7_9_g1c128dc [ 1683.080624] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 1683.083722] LNet: Accept secure, port 988 [ 1684.737258] Key type lgssc registered [ 1685.841421] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1692.481598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1709.024366] LustreError: 84401:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.156@tcp: failed processing log, type 1: rc = -5 [ 1740.770392] LustreError: 84401:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.156@tcp: failed processing log, type 4: rc = -110 [ 1770.530516] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1773.115078] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 1778.032856] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1783.730292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1783.901371] Lustre: *** cfs_fail_loc=706, val=0*** [ 1792.241410] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1792.308372] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1792.347191] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:161 [ 1795.169493] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 1797.615108] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1802.072942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1802.476664] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1802.531848] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:161 [ 1805.682535] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 1811.448622] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1815.976737] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1818.082668] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1818.084160] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1818.090917] LustreError: Skipped 1 previous similar message [ 1818.100508] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1818.103209] Lustre: Skipped 1 previous similar message [ 1820.129746] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1820.132684] Lustre: Skipped 1 previous similar message [ 1823.989253] Lustre: server umount lustre-OST0000 complete [ 1828.320897] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1828.325614] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1828.333155] Lustre: Skipped 1 previous similar message [ 1828.337896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1830.376192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1830.378567] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1832.403613] Lustre: server umount lustre-MDT0000 complete [ 1835.082107] LustreError: 84407:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761312186 with bad export cookie 9638323615570163583 [ 1835.087950] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1835.107674] LustreError: 84407:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1835.343713] Lustre: server umount lustre-MDT0001 complete [ 1843.213638] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 1845.663428] Key type lgssc unregistered [ 1845.932671] LNet: 88134:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1845.939781] LNet: Removed LNI 192.168.202.156@tcp [ 1846.588065] Key type .llcrypt unregistered [ 1846.589962] Key type ._llcrypt unregistered [ 1863.750772] alg: No test for adler32 (adler32-zlib) [ 1864.504465] Key type ._llcrypt registered [ 1864.506643] Key type .llcrypt registered [ 1864.582660] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 1865.402731] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1865.745846] Lustre: Lustre: Build Version: 2.15.7_9_g1c128dc [ 1865.887588] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 1865.896599] LNet: Accept secure, port 988 [ 1867.520142] Key type lgssc registered [ 1868.401095] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1874.205683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1878.265814] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 1883.455116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1883.680603] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1883.747601] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1886.595532] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 1889.251522] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1889.266746] LustreError: Skipped 2 previous similar messages [ 1892.740028] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1892.946510] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1892.959686] LustreError: Skipped 2 previous similar messages [ 1892.969589] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1896.513425] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 1898.467562] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1898.478399] LustreError: Skipped 2 previous similar messages [ 1899.518651] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:193 [ 1903.763347] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1903.782216] LustreError: Skipped 5 previous similar messages [ 1903.982336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1904.023727] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:193 [ 1907.336961] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 1919.460441] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1919.461230] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1919.477943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1924.356219] Lustre: server umount lustre-OST0000 complete [ 1929.701269] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1929.701286] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1929.701299] Lustre: Skipped 1 previous similar message [ 1929.702134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1929.702140] Lustre: Skipped 1 previous similar message [ 1933.414325] Lustre: server umount lustre-MDT0000 complete [ 1934.818559] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1934.828579] LustreError: Skipped 1 previous similar message [ 1936.059670] Lustre: server umount lustre-MDT0001 complete [ 1944.035215] Lustre: 92998:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761312289/real 1761312289] req@00000000c0104fe3 x1846869718217216/t0(0) o251->MGC192.168.202.156@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761312295 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1944.086214] Lustre: server umount MGS complete [ 1952.301753] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 09:25:02 (1761312302) [ 1978.251533] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 1985.922844] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 1992.496273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1996.393563] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2000.408369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2004.646845] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2009.018506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2009.068857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2011.605108] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 2017.968554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2018.011150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2018.173760] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2018.205992] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 2018.251930] Lustre: lustre-MDT0001: new disk, initializing [ 2018.302875] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2018.338720] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2018.342145] LustreError: 97699:0:(lod_dev.c:424:lod_sub_recovery_thread()) lustre-MDT0001-osd get update log failed: rc = -115 [ 2021.340516] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 2029.180696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2029.232400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2029.237901] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2029.392772] Lustre: lustre-OST0000: new disk, initializing [ 2029.401986] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2029.478235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2032.905314] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 2040.222267] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2040.360357] Lustre: 99137:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@00000000b2922b4b x1846869718221888/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1761312403 ref 2 fl Rpc:RQU/2/0 rc 0/-115 job:'' [ 2045.228857] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2045.262815] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2045.333613] Lustre: lustre-MDT0000: new disk, initializing [ 2045.495913] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2045.529501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2046.561168] Lustre: 99549:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@00000000ac1bd0c1 x1846869718225152/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1761312409 ref 2 fl Rpc:RQU/2/0 rc 0/-115 job:'' [ 2048.620383] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2048.638244] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2048.879538] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 2049.452423] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2074.092791] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2074.096353] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2074.109105] Lustre: Skipped 2 previous similar messages [ 2074.112656] Lustre: Skipped 1 previous similar message [ 2079.205387] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2079.208310] Lustre: Skipped 2 previous similar messages [ 2084.323759] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2084.326819] Lustre: Skipped 1 previous similar message [ 2085.344129] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2085.455146] Lustre: server umount lustre-OST0000 complete [ 2089.442400] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2089.444810] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2089.447672] Lustre: Skipped 1 previous similar message [ 2094.190734] Lustre: server umount lustre-MDT0000 complete [ 2094.565836] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2112.480190] Lustre: 100742:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761312457/real 1761312457] req@000000009816af9e x1846869718235776/t0(0) o251->MGC192.168.202.156@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761312463 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2112.508314] Lustre: server umount MGS complete [ 2112.512058] Lustre: Skipped 1 previous similar message [ 2120.671748] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 09:27:50 (1761312470) [ 2134.956149] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 2136.856605] Key type lgssc unregistered [ 2137.047831] LNet: 102148:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2137.054286] LNet: Removed LNI 192.168.202.156@tcp [ 2137.717740] Key type .llcrypt unregistered [ 2137.720965] Key type ._llcrypt unregistered [ 2149.935610] alg: No test for adler32 (adler32-zlib) [ 2150.685301] Key type ._llcrypt registered [ 2150.691963] Key type .llcrypt registered [ 2150.784337] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 2151.482395] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2151.726464] Lustre: Lustre: Build Version: 2.15.7_9_g1c128dc [ 2151.866517] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 2151.869723] LNet: Accept secure, port 988 [ 2153.480122] Key type lgssc registered [ 2154.186613] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2159.479725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2160.786237] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2160.829802] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2163.838480] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 2166.245333] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2169.684517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2169.907922] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2172.890583] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 2178.169332] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2182.030963] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2186.682398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2186.944314] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2190.015411] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 2192.051750] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 2195.488772] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2205.670768] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2205.671175] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2205.673683] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2205.687835] Lustre: Skipped 1 previous similar message [ 2206.728500] Lustre: server umount lustre-OST0000 complete [ 2210.785590] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2210.793328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2210.794333] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2210.804678] Lustre: Skipped 1 previous similar message [ 2210.807938] Lustre: Skipped 1 previous similar message [ 2215.795627] Lustre: server umount lustre-MDT0000 complete [ 2215.915255] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2215.923825] LustreError: Skipped 1 previous similar message [ 2218.905350] LustreError: 103395:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761312569 with bad export cookie 3242344933108396412 [ 2218.906949] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2218.919013] LustreError: 103395:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2219.142172] Lustre: server umount lustre-MDT0001 complete [ 2228.394460] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 2231.102245] Key type lgssc unregistered [ 2231.453231] LNet: 107067:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2231.495945] LNet: Removed LNI 192.168.202.156@tcp [ 2232.198250] Key type .llcrypt unregistered [ 2232.200506] Key type ._llcrypt unregistered [ 2244.769219] alg: No test for adler32 (adler32-zlib) [ 2245.523442] Key type ._llcrypt registered [ 2245.526297] Key type .llcrypt registered [ 2245.608994] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 2246.377921] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2246.781654] Lustre: Lustre: Build Version: 2.15.7_9_g1c128dc [ 2246.952630] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 2246.957957] LNet: Accept secure, port 988 [ 2248.608510] Key type lgssc registered [ 2249.697578] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2255.559514] LustreError: 108195:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 716 waking [ 2255.562948] LustreError: 108197:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 716 sleeping [ 2255.565296] LustreError: 108197:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 716 awake: rc=5000 [ 2255.568834] LustreError: 108197:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2255.573220] LustreError: 108197:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2255.617612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2256.928597] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2256.981521] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2261.988978] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2263.134206] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2263.421952] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2266.900164] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 2270.743526] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing lsmod [ 2272.700091] LustreError: 109504:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 716 sleeping [ 2272.706054] LustreError: 109505:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 716 waking [ 2272.711831] LustreError: 109504:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 716 awake: rc=4989 [ 2272.714636] LustreError: 109504:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2272.721084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2272.725909] LustreError: 109504:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2273.025512] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2278.116854] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2278.124132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2278.139855] Lustre: Skipped 1 previous similar message [ 2278.164079] Lustre: Skipped 2 previous similar messages [ 2281.326843] Lustre: server umount lustre-MDT0000 complete [ 2283.498134] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2283.514607] LustreError: Skipped 2 previous similar messages [ 2284.189531] LustreError: 109696:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761312635 with bad export cookie 10383086558464354388 [ 2284.199204] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2284.213558] LustreError: 109696:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2284.461818] Lustre: server umount lustre-MDT0001 complete [ 2293.728803] Lustre: 110100:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761312638/real 1761312638] req@0000000066e9f665 x1846870117721408/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761312644 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2293.922695] Lustre: server umount lustre-OST0000 complete [ 2301.061247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2301.462717] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2301.518518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2304.435856] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 2306.530573] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2310.557081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2310.809206] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2314.167191] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 2320.646981] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2324.840164] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2329.533346] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2329.824431] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2331.892736] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:65 [ 2333.347445] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 2340.633372] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2346.986861] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2346.992196] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2347.004326] Lustre: Skipped 1 previous similar message [ 2347.010960] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2350.564388] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2350.567226] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2350.576919] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2350.616300] Lustre: Skipped 1 previous similar message [ 2352.368358] Lustre: server umount lustre-OST0000 complete [ 2355.682852] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2355.687378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2355.693650] Lustre: Skipped 1 previous similar message [ 2360.800803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2360.803147] Lustre: Skipped 2 previous similar messages [ 2361.586622] Lustre: server umount lustre-MDT0000 complete [ 2364.380792] LustreError: 110555:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761312715 with bad export cookie 10383086558464355081 [ 2364.386365] LustreError: 110555:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2364.386837] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2373.107691] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 2375.764629] Key type lgssc unregistered [ 2375.987199] LNet: 114227:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2376.002701] LNet: Removed LNI 192.168.202.156@tcp [ 2376.565145] Key type .llcrypt unregistered [ 2376.566819] Key type ._llcrypt unregistered [ 2395.222468] alg: No test for adler32 (adler32-zlib) [ 2395.978330] Key type ._llcrypt registered [ 2395.984108] Key type .llcrypt registered [ 2396.109584] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 2397.362805] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2397.824065] Lustre: Lustre: Build Version: 2.15.7_9_g1c128dc [ 2398.049788] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 2398.058431] LNet: Accept secure, port 988 [ 2399.777119] Key type lgssc registered [ 2400.816446] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2407.545537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2408.888417] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2408.938041] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2412.230172] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 2414.049364] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2418.715935] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2418.975946] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2421.878527] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 2426.380183] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2429.911975] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2434.338364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2434.593496] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2437.058697] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 2439.744843] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 2443.093296] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2450.996110] Lustre: Setting parameter lustre-client.llite.some_wrong_param in log lustre-client [ 2453.048621] Lustre: Modifying parameter lustre-client.llite.some_wrong_param in log lustre-client [ 2456.047911] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2456.051606] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2456.060588] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2459.109118] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2459.114962] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2459.116520] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2459.137016] Lustre: Skipped 1 previous similar message [ 2464.226117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2469.352109] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2469.355033] Lustre: Skipped 2 previous similar messages [ 2470.377809] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2470.486467] Lustre: server umount lustre-OST0000 complete [ 2474.466708] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2474.472119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2474.476391] Lustre: Skipped 1 previous similar message [ 2479.812733] Lustre: server umount lustre-MDT0000 complete [ 2482.768268] LustreError: 115671:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761312833 with bad export cookie 9564051054297962108 [ 2482.771277] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2482.776707] LustreError: 115671:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2483.063742] Lustre: server umount lustre-MDT0001 complete [ 2491.994725] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 2494.514975] Key type lgssc unregistered [ 2494.803785] LNet: 119445:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2494.809317] LNet: Removed LNI 192.168.202.156@tcp [ 2495.383324] Key type .llcrypt unregistered [ 2495.385719] Key type ._llcrypt unregistered [ 2507.088592] alg: No test for adler32 (adler32-zlib) [ 2507.841451] Key type ._llcrypt registered [ 2507.844934] Key type .llcrypt registered [ 2507.966322] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 2508.622957] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2509.008610] Lustre: Lustre: Build Version: 2.15.7_9_g1c128dc [ 2509.171935] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 2509.175743] LNet: Accept secure, port 988 [ 2510.824130] Key type lgssc registered [ 2511.594073] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2518.463734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2519.814569] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2519.878040] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2523.066421] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 2525.165339] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2529.914888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2530.150684] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2534.073846] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 2539.995399] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2544.046887] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2548.458067] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2548.646480] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2550.705447] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 2551.497486] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 2556.600526] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2566.116962] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2566.117426] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2566.121030] LustreError: Skipped 1 previous similar message [ 2566.130205] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2566.134236] Lustre: Skipped 1 previous similar message [ 2571.238428] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2571.242442] Lustre: Skipped 1 previous similar message [ 2571.648052] Lustre: server umount lustre-OST0000 complete [ 2576.354073] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2576.362556] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2576.363819] Lustre: Skipped 1 previous similar message [ 2576.364393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2576.396540] Lustre: Skipped 1 previous similar message [ 2581.475338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2586.593324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2586.602170] Lustre: Skipped 2 previous similar messages [ 2588.128139] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2588.211209] Lustre: server umount lustre-MDT0000 complete [ 2590.713364] LustreError: 120692:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761312941 with bad export cookie 10150520446799844288 [ 2590.715919] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2590.720029] LustreError: 120692:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2590.936722] Lustre: server umount lustre-MDT0001 complete [ 2598.299977] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 2600.156778] Key type lgssc unregistered [ 2600.394711] LNet: 124371:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2600.407499] LNet: Removed LNI 192.168.202.156@tcp [ 2600.982471] Key type .llcrypt unregistered [ 2600.984448] Key type ._llcrypt unregistered [ 2615.857186] alg: No test for adler32 (adler32-zlib) [ 2616.616483] Key type ._llcrypt registered [ 2616.621938] Key type .llcrypt registered [ 2616.721732] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 2617.346715] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2617.710045] Lustre: Lustre: Build Version: 2.15.7_9_g1c128dc [ 2617.879227] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 2617.882318] LNet: Accept secure, port 988 [ 2619.504208] Key type lgssc registered [ 2620.300847] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2626.200988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2627.552963] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2627.590351] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2630.968694] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 2632.674582] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2637.395593] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2637.697653] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2640.798202] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 2646.305886] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2650.752748] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2655.816318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2656.094672] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2657.131555] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 2658.866556] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 2663.717521] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2672.084141] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash in log lustre-MDT0000 [ 2674.562489] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids in log lustre-MDT0000 [ 2674.566412] Lustre: Skipped 2 previous similar messages [ 2674.721759] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 2674.742224] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 2677.303158] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash in log lustre-MDT0000 [ 2677.305869] Lustre: Skipped 2 previous similar messages [ 2680.027601] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2680.030643] Lustre: Skipped 1 previous similar message [ 2683.736830] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids in log lustre-MDT0000 [ 2683.739824] Lustre: Skipped 2 previous similar messages [ 2688.540315] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.56@tcp 192.168.0.[2,10]@tcp [ 2693.093333] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2693.093896] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2693.098702] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2693.104185] LustreError: Skipped 1 previous similar message [ 2693.122987] Lustre: Skipped 1 previous similar message [ 2693.534086] Lustre: server umount lustre-OST0000 complete [ 2698.212605] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2698.238864] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2698.241640] Lustre: Skipped 1 previous similar message [ 2703.329746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2703.335584] Lustre: Skipped 1 previous similar message [ 2703.589028] Lustre: server umount lustre-MDT0000 complete [ 2707.444589] LustreError: 125812:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761313058 with bad export cookie 4751687431195638946 [ 2707.456446] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2707.472526] LustreError: 125812:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2707.940169] Lustre: server umount lustre-MDT0001 complete [ 2715.886983] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 2717.580871] Key type lgssc unregistered [ 2717.743533] LNet: 130546:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2717.749884] LNet: Removed LNI 192.168.202.156@tcp [ 2718.249515] Key type .llcrypt unregistered [ 2718.254230] Key type ._llcrypt unregistered [ 2735.655336] alg: No test for adler32 (adler32-zlib) [ 2736.406520] Key type ._llcrypt registered [ 2736.408107] Key type .llcrypt registered [ 2736.498584] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 2737.181336] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2737.438721] Lustre: Lustre: Build Version: 2.15.7_9_g1c128dc [ 2737.592069] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 2737.595319] LNet: Accept secure, port 988 [ 2739.200174] Key type lgssc registered [ 2740.010450] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2744.937346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2746.220091] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2746.257444] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2746.287032] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2746.293699] Lustre: lustre-MDT0000: nosquash_nids set to 2@gni 0@lo 192.168.202.56@tcp 192.168.0.[2,10]@tcp [ 2748.968555] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 2751.458286] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2753.712193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2753.887868] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2753.904263] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.56@tcp 192.168.0.[2,10]@tcp [ 2753.944409] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2756.813682] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 2761.852893] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2765.892813] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2771.281571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2771.518209] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2774.954758] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 2776.614927] Lustre: lustre-OST0000: deleting orphan objects from 0x0:167 to 0x0:193 [ 2780.775258] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2791.907703] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2791.919236] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2791.937966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2793.960527] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2793.967104] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2793.983130] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2794.030461] Lustre: Skipped 1 previous similar message [ 2797.542863] Lustre: server umount lustre-OST0000 complete [ 2804.199870] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2804.202161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2804.213844] Lustre: Skipped 1 previous similar message [ 2807.204032] Lustre: server umount lustre-MDT0000 complete [ 2809.315053] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2809.323678] LustreError: Skipped 1 previous similar message [ 2810.387537] LustreError: 134106:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761313161 with bad export cookie 9466363554228410424 [ 2810.391310] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2810.394500] LustreError: 134106:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2810.660402] Lustre: server umount lustre-MDT0001 complete [ 2818.669757] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 2820.600875] Key type lgssc unregistered [ 2820.808421] LNet: 135859:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2820.812879] LNet: Removed LNI 192.168.202.156@tcp [ 2821.489927] Key type .llcrypt unregistered [ 2821.491410] Key type ._llcrypt unregistered [ 2845.906991] alg: No test for adler32 (adler32-zlib) [ 2846.660481] Key type ._llcrypt registered [ 2846.661839] Key type .llcrypt registered [ 2846.819903] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 2857.184302] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 2857.818710] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2858.409317] Lustre: Lustre: Build Version: 2.15.7_9_g1c128dc [ 2858.591552] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 2858.597944] LNet: Accept secure, port 988 [ 2860.248141] Key type lgssc registered [ 2861.198679] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2868.103476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2873.156957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2877.696806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2882.205885] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2887.311424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2887.382271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2888.528371] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2888.566550] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2888.687615] Lustre: lustre-MDT0000: new disk, initializing [ 2888.793118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2888.813198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2892.263506] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 2901.173125] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2901.223473] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2901.293351] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2901.326230] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 2901.329449] Lustre: Skipped 1 previous similar message [ 2901.377971] Lustre: lustre-MDT0001: new disk, initializing [ 2901.478869] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2901.523509] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2901.537762] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2905.768886] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 2914.251554] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2919.636287] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2925.980510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2926.051682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2926.063190] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2926.320723] Lustre: lustre-OST0000: new disk, initializing [ 2926.326643] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2926.435296] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2930.234460] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 2934.309912] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2934.325977] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2937.994032] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2941.606663] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2941.787339] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2945.313587] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2945.490872] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2949.606121] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2949.610382] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2949.634817] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2950.114380] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2950.127577] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2950.153978] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2950.157018] Lustre: Skipped 1 previous similar message [ 2953.571077] Lustre: server umount lustre-OST0000 complete [ 2957.796903] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2957.801847] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2957.820241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2960.355246] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2960.355798] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2960.370053] Lustre: Skipped 1 previous similar message [ 2962.796202] Lustre: server umount lustre-MDT0000 complete [ 2965.470475] LustreError: 139372:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761313316 with bad export cookie 387459621120299179 [ 2965.473714] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2965.481280] LustreError: 139372:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2965.719626] Lustre: server umount lustre-MDT0001 complete [ 2972.529142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2972.837832] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2972.904574] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2976.047525] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 2978.274979] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2982.633465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2983.028222] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2986.228846] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 2991.902645] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2995.930749] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3000.914842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3001.188877] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3004.496591] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 3011.442758] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3015.105608] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3015.270723] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3018.647709] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3018.826470] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3026.236908] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3026.310723] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3026.318562] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 3026.424154] Lustre: lustre-OST0001: new disk, initializing [ 3026.426963] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 3026.513729] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3030.437873] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 3033.678846] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3033.684992] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3037.179824] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 3037.361901] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3041.055345] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 3041.253248] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3046.039598] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9047055a0000.ost_server_uuid 40 [ 3047.180113] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9047055a0000.ost_server_uuid in FULL state after 0 sec [ 3054.564905] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3054.568172] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3054.573870] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3054.584035] Lustre: Skipped 1 previous similar message [ 3058.812858] Lustre: server umount lustre-OST0001 complete [ 3059.684843] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3059.693789] LustreError: Skipped 1 previous similar message [ 3063.783527] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3063.790531] LustreError: Skipped 1 previous similar message [ 3063.805180] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3063.816271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3063.819224] Lustre: Skipped 1 previous similar message [ 3064.802596] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3064.816845] LustreError: Skipped 1 previous similar message [ 3068.310401] Lustre: server umount lustre-OST0000 complete [ 3075.045520] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3075.052946] LustreError: Skipped 1 previous similar message [ 3080.160904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3080.165231] Lustre: Skipped 5 previous similar messages [ 3086.308306] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3086.425176] Lustre: server umount lustre-MDT0000 complete [ 3089.521419] LustreError: 147271:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761313440 with bad export cookie 387459621120300033 [ 3089.528668] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3089.530530] LustreError: 147271:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3108.809770] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 3111.105388] Key type lgssc unregistered [ 3111.343357] LNet: 150386:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3111.388763] LNet: Removed LNI 192.168.202.156@tcp [ 3112.120546] Key type .llcrypt unregistered [ 3112.122360] Key type ._llcrypt unregistered [ 3138.162231] alg: No test for adler32 (adler32-zlib) [ 3138.908400] Key type ._llcrypt registered [ 3138.910201] Key type .llcrypt registered [ 3138.999820] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 3150.640364] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 3151.311557] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3151.914912] Lustre: Lustre: Build Version: 2.15.7_9_g1c128dc [ 3152.078478] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 3152.082176] LNet: Accept secure, port 988 [ 3153.752179] Key type lgssc registered [ 3154.753754] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3163.076615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3168.597985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3174.428667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3180.352206] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3187.238565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3187.332270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3188.577111] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3188.610753] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3188.711982] Lustre: lustre-MDT0000: new disk, initializing [ 3188.796087] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3188.821420] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3191.569393] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 3199.675262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3199.729448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3199.780454] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3199.809288] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3199.811431] Lustre: Skipped 1 previous similar message [ 3199.867793] Lustre: lustre-MDT0001: new disk, initializing [ 3199.946826] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3199.979213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3199.984567] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3203.328790] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 3210.779988] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3214.895452] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3219.917845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3219.972892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3219.977780] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3220.132754] Lustre: lustre-OST0000: new disk, initializing [ 3220.137490] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3220.206031] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3223.496681] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 3229.248090] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3229.252735] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3230.423318] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3234.215939] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3234.406541] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3238.421633] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3238.626839] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3245.028226] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3245.031415] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3245.034939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3245.067276] LustreError: Skipped 1 previous similar message [ 3245.072080] Lustre: Skipped 1 previous similar message [ 3246.667899] Lustre: server umount lustre-OST0000 complete [ 3250.145695] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3250.159596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3250.173129] Lustre: Skipped 1 previous similar message [ 3255.267179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3255.269730] Lustre: Skipped 1 previous similar message [ 3256.049811] Lustre: server umount lustre-MDT0000 complete [ 3259.504685] LustreError: 154474:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761313610 with bad export cookie 905227069095911595 [ 3259.508226] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3259.538502] LustreError: 154474:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3259.845727] Lustre: server umount lustre-MDT0001 complete [ 3267.062886] Lustre: DEBUG MARKER: == conf-sanity test 47: server restart does not make client loss lru_resize settings ========================================================== 09:46:57 (1761313617) [ 3280.264781] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 3288.569460] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 3299.848619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3307.773242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3313.385670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3319.900104] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3326.067690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3326.146202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3326.457855] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3326.495593] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3326.575694] Lustre: lustre-MDT0000: new disk, initializing [ 3326.645855] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3326.664335] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3331.061171] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 3340.106103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3340.186296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3340.246842] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3340.286657] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3340.290914] Lustre: Skipped 1 previous similar message [ 3340.354834] Lustre: lustre-MDT0001: new disk, initializing [ 3340.441074] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3340.480730] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3340.499038] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3343.978282] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 3351.500532] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3355.309814] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3359.005040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3359.037143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3359.041943] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3359.198786] Lustre: lustre-OST0000: new disk, initializing [ 3359.203622] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3359.267364] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3361.873747] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 3366.929451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3366.935158] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3368.374851] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3371.127054] Lustre: Failing over lustre-OST0000 [ 3371.239978] Lustre: server umount lustre-OST0000 complete [ 3378.144235] Lustre: 152705:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761313722/real 1761313722] req@000000009d3fff07 x1846871067753216/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761313729 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3378.145719] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3378.167053] Lustre: 152705:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 3378.172214] Lustre: Skipped 2 previous similar messages [ 3382.752248] Lustre: 152703:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761313726/real 1761313726] req@00000000c101b5b3 x1846871067753408/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1761313733 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 3382.786188] Lustre: 152703:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3384.292247] Lustre: 152703:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761313728/real 1761313728] req@000000005e301bb4 x1846871067753856/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761313735 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3389.341854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3389.702388] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3389.723833] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3394.044592] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 3395.817195] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3397.371359] Lustre: Failing over lustre-MDT0000 [ 3397.397204] LustreError: 152701:0:(import.c:692:ptlrpc_connect_import_locked()) can't connect to a closed import [ 3397.567258] Lustre: server umount lustre-MDT0000 complete [ 3398.289724] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3398.309503] Lustre: lustre-OST0000: Recovery over after 0:03, of 3 clients 3 recovered and 0 were evicted. [ 3398.309671] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to (at 0@lo) [ 3401.188353] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3401.194568] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3401.206074] Lustre: Skipped 2 previous similar messages [ 3401.216158] LustreError: Skipped 2 previous similar messages [ 3403.381669] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3405.469335] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3406.818723] Lustre: 161476:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761313750/real 1761313750] req@0000000010b31788 x1846871067758784/t0(0) o101->MGC192.168.202.156@tcp@0@lo:26/25 lens 328/344 e 0 to 1 dl 1761313757 ref 2 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 3406.832678] Lustre: 161476:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3406.836523] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3410.544465] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3410.581945] LustreError: Skipped 3 previous similar messages [ 3416.505534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3421.812414] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3421.846962] LustreError: Skipped 11 previous similar messages [ 3423.212598] Lustre: Evicted from MGS (at 192.168.202.156@tcp) after server handle changed from 0xc9003392687a401 to 0xc9003392687a8bc [ 3423.232100] Lustre: MGC192.168.202.156@tcp: Connection restored to (at 0@lo) [ 3423.501178] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3423.574111] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3423.614840] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3426.928235] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3427.333876] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 3428.850900] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 3428.907469] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 3429.000689] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 3439.078458] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3439.091906] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3439.095536] Lustre: Skipped 1 previous similar message [ 3442.591684] Lustre: server umount lustre-OST0000 complete [ 3449.313402] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3449.337510] Lustre: Skipped 1 previous similar message [ 3449.341968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3449.353084] Lustre: Skipped 1 previous similar message [ 3454.433483] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3454.449141] LustreError: Skipped 4 previous similar messages [ 3456.016360] LustreError: 161463:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761313806 with bad export cookie 905227069095913660 [ 3456.020766] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3467.797566] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 3470.568780] Key type lgssc unregistered [ 3470.919769] LNet: 167323:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3470.926485] LNet: Removed LNI 192.168.202.156@tcp [ 3471.679783] Key type .llcrypt unregistered [ 3471.681437] Key type ._llcrypt unregistered [ 3492.796676] alg: No test for adler32 (adler32-zlib) [ 3493.562053] Key type ._llcrypt registered [ 3493.564467] Key type .llcrypt registered [ 3493.690665] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 3494.765387] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3495.180993] Lustre: Lustre: Build Version: 2.15.7_9_g1c128dc [ 3495.337273] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 3495.342557] LNet: Accept secure, port 988 [ 3496.968180] Key type lgssc registered [ 3497.973413] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3504.620775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3506.229716] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3506.279817] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3510.247613] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 3511.777750] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3516.902463] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3519.560544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3520.256452] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3525.343418] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 3535.081580] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3540.788598] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3547.395920] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3547.857295] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3551.558387] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 3556.097936] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:65 [ 3558.018934] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3577.488994] Lustre: lustre-MDT0000: Client f7288ca2-5897-4376-b209-8bc5ac8615af (at 192.168.202.56@tcp) reconnecting [ 3587.243177] Lustre: lustre-MDT0000: Client f7288ca2-5897-4376-b209-8bc5ac8615af (at 192.168.202.56@tcp) reconnecting [ 3635.859945] Lustre: lustre-MDT0000: Client f7288ca2-5897-4376-b209-8bc5ac8615af (at 192.168.202.56@tcp) reconnecting [ 3685.007110] Lustre: lustre-MDT0000: Client f7288ca2-5897-4376-b209-8bc5ac8615af (at 192.168.202.56@tcp) reconnecting [ 3734.207877] Lustre: lustre-MDT0000: Client f7288ca2-5897-4376-b209-8bc5ac8615af (at 192.168.202.56@tcp) reconnecting [ 3780.239381] Lustre: lustre-MDT0000: Client f7288ca2-5897-4376-b209-8bc5ac8615af (at 192.168.202.56@tcp) reconnecting [ 3829.409218] Lustre: lustre-MDT0000: Client f7288ca2-5897-4376-b209-8bc5ac8615af (at 192.168.202.56@tcp) reconnecting [ 3877.549743] Lustre: lustre-MDT0000: Client f7288ca2-5897-4376-b209-8bc5ac8615af (at 192.168.202.56@tcp) reconnecting [ 3975.897388] Lustre: lustre-MDT0000: Client f7288ca2-5897-4376-b209-8bc5ac8615af (at 192.168.202.56@tcp) reconnecting [ 3975.916821] Lustre: Skipped 1 previous similar message [ 4119.260825] Lustre: lustre-MDT0000: Client f7288ca2-5897-4376-b209-8bc5ac8615af (at 192.168.202.56@tcp) reconnecting [ 4119.280927] Lustre: Skipped 2 previous similar messages [ 4401.821640] Lustre: lustre-MDT0000: Client f7288ca2-5897-4376-b209-8bc5ac8615af (at 192.168.202.56@tcp) reconnecting [ 4401.833692] Lustre: Skipped 12 previous similar messages [ 4934.406918] Lustre: lustre-MDT0000: Client f7288ca2-5897-4376-b209-8bc5ac8615af (at 192.168.202.56@tcp) reconnecting [ 4934.423838] Lustre: Skipped 11 previous similar messages [ 5381.332761] Lustre: Failing over lustre-MDT0000 [ 5381.509316] Lustre: server umount lustre-MDT0000 complete [ 5382.119054] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5382.125722] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5382.132468] Lustre: Skipped 2 previous similar messages [ 5382.144431] LustreError: Skipped 1 previous similar message [ 5383.284221] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5387.233884] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5387.239050] LustreError: Skipped 2 previous similar messages [ 5389.216172] Lustre: 168108:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761315733/real 1761315733] req@00000000246f061b x1846871427977088/t0(0) o400->MGC192.168.202.156@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761315740 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 5389.257830] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5389.284797] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5389.308916] LustreError: Skipped 3 previous similar messages [ 5394.401991] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5394.410328] LustreError: Skipped 3 previous similar messages [ 5399.362928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5405.665601] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5405.685827] LustreError: Skipped 11 previous similar messages [ 5406.696856] Lustre: Evicted from MGS (at 192.168.202.156@tcp) after server handle changed from 0x96b1417197ec4824 to 0x96b1417197eeb44c [ 5406.718933] Lustre: MGC192.168.202.156@tcp: Connection restored to (at 0@lo) [ 5406.839603] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.56@tcp (not set up) [ 5406.959268] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5406.997920] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5410.516912] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 5411.954822] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5412.334410] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 5412.348620] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 5412.389754] Lustre: lustre-OST0000: deleting orphan objects from 0x0:68 to 0x0:97 [ 5418.222249] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5419.882283] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5437.925900] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5437.929750] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5437.939241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5437.941996] Lustre: Skipped 1 previous similar message [ 5443.043239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5448.165607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5448.168543] Lustre: Skipped 2 previous similar messages [ 5450.208384] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 5450.320462] Lustre: server umount lustre-MDT0000 complete [ 5453.286221] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5453.304323] LustreError: Skipped 5 previous similar messages [ 5453.974520] LustreError: 168778:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761315804 with bad export cookie 10858532132703089740 [ 5453.980348] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5453.999761] LustreError: 168778:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5458.401203] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5458.408424] Lustre: Skipped 2 previous similar messages [ 5458.418970] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5460.406908] Lustre: server umount lustre-MDT0001 complete [ 5478.369193] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 5478.645361] Lustre: server umount lustre-OST0000 complete [ 5487.023787] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 5494.155934] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 5502.236944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5507.641394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5513.033798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5518.269366] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5523.521159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5523.581064] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5523.760949] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5523.797617] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5523.890390] Lustre: lustre-MDT0000: new disk, initializing [ 5524.002841] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5524.020442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5527.807881] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 5537.474167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5537.524437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5537.621971] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5537.663673] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 5537.667361] Lustre: Skipped 1 previous similar message [ 5537.739178] Lustre: lustre-MDT0001: new disk, initializing [ 5537.826544] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5537.868749] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5537.886978] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5541.517162] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 5549.889392] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5555.713735] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5562.076875] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5562.120255] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5562.124571] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 5562.324911] Lustre: lustre-OST0000: new disk, initializing [ 5562.333843] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5562.439097] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5566.239541] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 5570.636262] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5570.642579] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5574.409574] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5578.450138] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 5578.711388] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5582.767345] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 5583.000499] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5586.410724] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5586.417697] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5586.428333] LustreError: Skipped 1 previous similar message [ 5586.437743] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5586.437765] Lustre: Skipped 1 previous similar message [ 5586.498361] Lustre: Skipped 1 previous similar message [ 5591.353435] Lustre: server umount lustre-OST0000 complete [ 5596.643557] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5596.676578] Lustre: Skipped 1 previous similar message [ 5600.695220] Lustre: server umount lustre-MDT0000 complete [ 5601.761637] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5601.771064] LustreError: Skipped 3 previous similar messages [ 5603.901308] LustreError: 175932:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761315954 with bad export cookie 10858532132703090461 [ 5603.904245] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5603.906045] LustreError: 175932:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5611.921358] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 10:26:01 (1761315961) [ 5626.105804] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 5633.972802] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 5642.887794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5648.706519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5653.578562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5659.170530] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5664.710794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5664.781213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5665.001333] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5665.049298] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5665.136476] Lustre: lustre-MDT0000: new disk, initializing [ 5665.214379] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5665.233883] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5668.304405] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 5677.437533] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5677.499217] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5677.602709] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5677.657600] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 5677.660688] Lustre: Skipped 1 previous similar message [ 5677.715857] Lustre: lustre-MDT0001: new disk, initializing [ 5677.874329] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5677.919266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5677.929067] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5681.416761] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 5689.325961] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5693.969088] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5699.808793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5699.876806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5699.881819] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 5700.107310] Lustre: lustre-OST0000: new disk, initializing [ 5700.109941] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5700.205308] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5703.869102] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 5708.380762] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5708.388727] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5712.532801] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5724.130599] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5724.131997] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5724.146878] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5724.165344] Lustre: Skipped 3 previous similar messages [ 5724.167381] Lustre: Skipped 1 previous similar message [ 5725.983153] Lustre: server umount lustre-OST0000 complete [ 5725.986090] Lustre: Skipped 1 previous similar message [ 5734.370925] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5734.374508] LustreError: Skipped 1 previous similar message [ 5744.096182] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 5745.634494] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5745.651717] LustreError: Skipped 3 previous similar messages [ 5747.876902] LustreError: 182919:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761316098 with bad export cookie 10858532132703091315 [ 5747.877940] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5747.891109] LustreError: 182919:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5756.067236] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 10:28:26 (1761316106) [ 5769.728423] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 5779.218844] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 5790.756239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5797.175134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5802.712532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5807.916839] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5814.159534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5814.227387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5814.479424] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5814.522891] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5814.650329] Lustre: lustre-MDT0000: new disk, initializing [ 5814.757539] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5814.800697] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5818.949791] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 5829.185935] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5829.255967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5829.332657] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5829.540110] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5833.105343] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 5842.318733] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5847.387093] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5853.506209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5853.575822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5853.580331] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 5853.753357] Lustre: lustre-OST0000: new disk, initializing [ 5853.755454] Lustre: Skipped 1 previous similar message [ 5853.758464] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5853.761495] Lustre: Skipped 2 previous similar messages [ 5858.207961] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 5861.443282] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5861.454625] Lustre: Skipped 1 previous similar message [ 5861.456866] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5867.784314] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5873.719657] Lustre: server umount lustre-OST0000 complete [ 5873.721283] Lustre: Skipped 2 previous similar messages [ 5877.221209] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5877.223373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5877.235433] Lustre: Skipped 3 previous similar messages [ 5877.253236] Lustre: Skipped 8 previous similar messages [ 5882.848386] Lustre: 192546:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761316227/real 1761316227] req@00000000d5a83adc x1846871428045312/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761316233 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 5884.388229] Lustre: 168111:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761316228/real 1761316228] req@00000000bd9c7c10 x1846871428045376/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1761316235 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 5884.432530] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5886.762172] LustreError: 189386:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761316237 with bad export cookie 10858532132703092400 [ 5886.765197] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5886.774629] LustreError: 189386:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5898.368459] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 5901.203551] Key type lgssc unregistered [ 5901.514074] LNet: 193351:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5901.521986] LNet: Removed LNI 192.168.202.156@tcp [ 5902.388572] Key type .llcrypt unregistered [ 5902.390659] Key type ._llcrypt unregistered [ 5924.767077] alg: No test for adler32 (adler32-zlib) [ 5925.520415] Key type ._llcrypt registered [ 5925.522044] Key type .llcrypt registered [ 5925.612391] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 5926.427653] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5927.085870] Lustre: Lustre: Build Version: 2.15.7_9_g1c128dc [ 5927.339737] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 5927.344052] LNet: Accept secure, port 988 [ 5929.048133] Key type lgssc registered [ 5929.884478] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5937.292824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5938.851360] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5938.905875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5942.946955] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 5944.295390] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5949.413152] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5951.139183] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5951.361588] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5955.391696] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 5962.869317] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5969.009677] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5975.820949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5976.222212] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5980.682473] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 5988.041746] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5989.490114] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 6010.345723] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6010.350946] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6010.359402] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6010.361715] Lustre: Skipped 1 previous similar message [ 6014.886349] Lustre: server umount lustre-OST0000 complete [ 6020.580457] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6020.586904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6020.598745] Lustre: Skipped 1 previous similar message [ 6020.600403] Lustre: Skipped 1 previous similar message [ 6024.481413] Lustre: server umount lustre-MDT0000 complete [ 6025.717857] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6025.733589] LustreError: Skipped 1 previous similar message [ 6028.371890] LustreError: 194793:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761316379 with bad export cookie 6548949947275793541 [ 6028.373808] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6028.436919] LustreError: 194793:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6029.392417] Lustre: server umount lustre-MDT0001 complete [ 6042.964431] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 6045.826263] Key type lgssc unregistered [ 6046.165117] LNet: 198494:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6046.187824] LNet: Removed LNI 192.168.202.156@tcp [ 6047.026832] Key type .llcrypt unregistered [ 6047.028638] Key type ._llcrypt unregistered [ 6069.331552] alg: No test for adler32 (adler32-zlib) [ 6070.100141] Key type ._llcrypt registered [ 6070.108542] Key type .llcrypt registered [ 6070.260225] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 6071.694187] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6072.010322] Lustre: Lustre: Build Version: 2.15.7_9_g1c128dc [ 6072.187578] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 6072.190728] LNet: Accept secure, port 988 [ 6073.904195] Key type lgssc registered [ 6075.070530] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6082.739279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6084.121472] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6084.166624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6087.842573] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 6089.191685] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6094.315120] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6096.153587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6096.781993] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6100.894369] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 6108.449829] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6114.316940] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6119.821876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6120.102705] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6123.959581] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 6130.368757] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6134.452582] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 6141.081851] Lustre: lustre-OST0000: Not available for connect from 192.168.202.56@tcp (stopping) [ 6141.921361] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6141.931051] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6141.949938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6145.000904] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6145.003957] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6145.005325] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6145.015165] Lustre: Skipped 1 previous similar message [ 6146.526442] Lustre: server umount lustre-OST0000 complete [ 6153.826578] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff904711af5800.ost_server_uuid 40 [ 6155.221392] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6168.863982] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6168.899093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6168.902378] Lustre: Skipped 1 previous similar message [ 6171.874448] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6175.144503] Lustre: server umount lustre-MDT0000 complete [ 6177.252230] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6177.262351] LustreError: Skipped 1 previous similar message [ 6179.179382] LustreError: 199940:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761316530 with bad export cookie 13259658013373392351 [ 6179.198559] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6179.464895] Lustre: server umount lustre-MDT0001 complete [ 6187.938662] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 10:35:37 (1761316537) [ 6193.673124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6194.072368] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6194.165794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6197.940312] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 6204.387067] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6204.401521] LustreError: Skipped 1 previous similar message [ 6205.487413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6205.792606] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6209.635858] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 6216.510397] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6221.606993] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6226.707538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6226.972494] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6230.206413] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 6236.510089] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6241.526479] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6241.583628] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6241.588892] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 6241.699064] Lustre: lustre-OST0001: new disk, initializing [ 6241.702254] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 6244.859035] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 6245.162329] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 6250.059632] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6250.065937] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6253.233879] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6255.586608] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6255.609758] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6255.616962] Lustre: Skipped 2 previous similar messages [ 6260.194285] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6263.924642] Lustre: lustre-OST0000: Not available for connect from 192.168.202.56@tcp (stopping) [ 6263.929406] Lustre: Skipped 3 previous similar messages [ 6269.408335] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 6269.468662] Lustre: server umount lustre-OST0000 complete [ 6270.950946] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6270.960324] LustreError: Skipped 1 previous similar message [ 6274.246611] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 6274.389387] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6277.741210] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 6277.903833] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6291.428275] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6291.431624] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6291.437617] LustreError: Skipped 1 previous similar message [ 6291.438335] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6291.443302] Lustre: Skipped 2 previous similar messages [ 6291.445493] Lustre: Skipped 3 previous similar messages [ 6296.090357] Lustre: server umount lustre-OST0001 complete [ 6305.460565] Lustre: server umount lustre-MDT0000 complete [ 6306.788092] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6306.795059] LustreError: Skipped 12 previous similar messages [ 6308.683922] LustreError: 208369:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761316659 with bad export cookie 13259658013373393415 [ 6308.687037] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6308.710605] LustreError: 208369:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 6314.545558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6317.804964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6321.461800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6324.579418] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6330.015473] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 10:38:00 (1761316680) [ 6335.400736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6335.639756] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6335.670832] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6335.881973] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6335.886507] Lustre: Skipped 1 previous similar message [ 6340.104699] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 6347.752893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6347.811136] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6347.844228] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6351.679641] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 6357.903256] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6362.590652] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6368.511329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6368.710103] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6368.894722] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6368.907299] Lustre: Skipped 1 previous similar message [ 6371.869780] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 6378.097196] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 6378.635931] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6385.668993] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6385.788323] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6390.810561] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 6397.162241] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6403.048884] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6403.073040] Lustre: Skipped 2 previous similar messages [ 6403.076147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6403.078896] Lustre: Skipped 3 previous similar messages [ 6414.304210] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 6414.345452] Lustre: server umount lustre-OST0000 complete [ 6414.347468] Lustre: Skipped 1 previous similar message [ 6418.406487] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6418.428567] LustreError: Skipped 3 previous similar messages [ 6428.645181] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6438.881503] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6438.899944] Lustre: Skipped 3 previous similar messages [ 6446.526358] LustreError: 210359:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761316797 with bad export cookie 13259658013373394794 [ 6446.532048] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6446.541863] LustreError: 210359:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6446.788909] Lustre: server umount lustre-MDT0001 complete [ 6446.790556] Lustre: Skipped 2 previous similar messages [ 6451.273928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6454.278993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6457.731699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6461.347076] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6467.499683] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 10:40:17 (1761316817) [ 6480.570415] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 6489.041266] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 6499.019786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6505.014448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6510.435202] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6516.543574] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6523.590929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6523.679716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6523.961905] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6524.002167] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 6524.094762] Lustre: lustre-MDT0000: new disk, initializing [ 6524.201867] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6524.205059] Lustre: Skipped 1 previous similar message [ 6524.229153] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6528.438307] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 6538.321746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6538.381519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6538.462129] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6538.513926] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 6538.517145] Lustre: Skipped 1 previous similar message [ 6538.584947] Lustre: lustre-MDT0001: new disk, initializing [ 6538.717565] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6538.740461] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6543.444458] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 6554.183705] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6560.102099] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6566.131569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6566.190848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6566.196346] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 6566.546700] Lustre: lustre-OST0000: new disk, initializing [ 6566.553531] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 6570.877144] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 6574.693262] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6574.718255] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6579.131960] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6583.073465] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 6583.259907] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6587.050805] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 6587.286654] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6590.433976] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6590.437655] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6590.450888] LustreError: Skipped 1 previous similar message [ 6590.456185] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6590.482281] Lustre: Skipped 2 previous similar messages [ 6590.494827] Lustre: Skipped 15 previous similar messages [ 6603.232136] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 6603.355947] Lustre: server umount lustre-OST0000 complete [ 6610.401570] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6616.037859] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6616.064135] LustreError: Skipped 11 previous similar messages [ 6617.774393] LustreError: 218600:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761316968 with bad export cookie 13259658013373396320 [ 6617.788499] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6617.804563] LustreError: 218600:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6627.899216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6633.101433] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 6643.245910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6647.759674] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 6655.220502] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6660.827143] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6667.517597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6667.969781] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6667.976981] Lustre: Skipped 4 previous similar messages [ 6672.558409] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 6679.556879] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6683.373590] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 6683.553854] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6688.625475] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 6688.927342] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6694.886629] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6703.944663] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 6708.119846] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 6735.282561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6739.086537] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 6745.149666] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6746.549098] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6747.874938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6747.893696] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6747.903150] Lustre: lustre-OST0000: deleting orphan objects from 0x0:34 to 0x0:65 [ 6747.905179] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.156@tcp (at 0@lo) [ 6747.944028] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:65 [ 6753.252635] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6753.256681] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6753.272364] Lustre: Skipped 5 previous similar messages [ 6753.296513] Lustre: Skipped 8 previous similar messages [ 6755.661646] Lustre: server umount lustre-OST0000 complete [ 6755.664930] Lustre: Skipped 3 previous similar messages [ 6761.441641] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6761.446346] LustreError: Skipped 3 previous similar messages [ 6768.364880] LustreError: 222940:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761317119 with bad export cookie 13259658013373397174 [ 6768.371681] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6768.374716] LustreError: 222940:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6775.818105] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 10:45:25 (1761317125) [ 6781.485307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6785.083909] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 6791.329946] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6794.513356] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 6799.511867] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6803.492295] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6808.054256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6811.872509] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 6818.021497] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6821.919938] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 6822.106812] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6826.327850] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 6826.615710] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6832.510321] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6832.592658] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6832.605574] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 6832.733385] Lustre: lustre-OST0001: new disk, initializing [ 6832.745372] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 6836.859810] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 6842.425421] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6842.442784] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6845.599676] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6849.851852] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 6850.082571] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6854.402369] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 6854.593019] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6867.838451] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 6868.091394] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6871.821350] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 6872.031553] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6872.416766] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 192.168.202.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6872.424591] LustreError: Skipped 10 previous similar messages [ 6899.648336] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6903.470613] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 6905.324839] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6905.339041] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to (at 0@lo) [ 6905.339210] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6905.346872] Lustre: Skipped 1 previous similar message [ 6909.795435] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6911.155545] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6915.567922] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6952.084720] LustreError: 229138:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761317303 with bad export cookie 13259658013373398588 [ 6952.088535] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6952.092017] LustreError: 229138:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6963.097033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6966.381126] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6969.648729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6972.741253] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6977.894252] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 10:48:47 (1761317327) [ 6983.352902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6983.575699] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6983.598597] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6983.732981] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6983.736815] Lustre: Skipped 6 previous similar messages [ 6988.710327] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 6998.787763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6999.000496] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6999.041375] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 7004.059499] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 7011.931775] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7018.024392] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7023.899265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7024.138648] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7028.098735] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 7034.248395] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7042.747849] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7042.851217] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7046.582722] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 7052.218606] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7053.587457] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7057.523818] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 7057.730661] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7061.591607] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 7061.841934] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7067.456384] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9047185a5800.ost_server_uuid 40 [ 7068.879126] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9047185a5800.ost_server_uuid in IDLE state after 0 sec [ 7069.776805] Lustre: Permanently deactivating lustre-OST0001 [ 7069.792548] Lustre: Setting parameter lustre-OST0001-osc.osc.active in log lustre-client [ 7073.557610] Lustre: Permanently reactivating lustre-OST0001 [ 7082.472361] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7082.477064] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7082.481866] Lustre: Skipped 11 previous similar messages [ 7082.487897] Lustre: Skipped 17 previous similar messages [ 7084.035126] Lustre: server umount lustre-OST0001 complete [ 7084.041426] Lustre: Skipped 7 previous similar messages [ 7108.624100] LustreError: 241012:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761317459 with bad export cookie 13259658013373399862 [ 7108.635680] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7114.066111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7117.388040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7120.283082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7122.976055] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7127.182822] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 10:51:17 (1761317477) [ 7127.836783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7132.310816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7132.562726] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7132.584937] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 7132.588212] Lustre: Skipped 5 previous similar messages [ 7135.958652] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 7141.985317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7142.052575] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7145.086588] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 7150.432493] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7154.925469] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7159.856212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7159.929449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7160.094924] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7160.108954] Lustre: Permanently deactivating lustre-OST0000 [ 7163.894454] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 7165.990910] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7171.492694] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7177.709314] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7177.944519] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7181.770601] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 7183.990709] Lustre: lustre-OST0001: deleting orphan objects from 0x0:67 to 0x0:97 [ 7188.677228] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7190.203322] Lustre: Permanently reactivating lustre-OST0000 [ 7200.323819] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7200.332470] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7200.345566] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to (at 0@lo) [ 7200.375470] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 7200.411220] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 7225.322339] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7225.325849] LustreError: Skipped 9 previous similar messages [ 7247.328142] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 7265.248255] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 7268.958500] LustreError: 244904:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761317619 with bad export cookie 13259658013373401465 [ 7268.961963] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7269.022181] LustreError: 244904:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 7269.024594] LustreError: 199270:0:(import.c:702:ptlrpc_connect_import_locked()) already connecting [ 7281.198532] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 7284.644956] Key type lgssc unregistered [ 7284.920659] LNet: 249883:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7284.931068] LNet: Removed LNI 192.168.202.156@tcp [ 7285.776639] Key type .llcrypt unregistered [ 7285.779636] Key type ._llcrypt unregistered [ 7305.698832] alg: No test for adler32 (adler32-zlib) [ 7306.448450] Key type ._llcrypt registered [ 7306.450086] Key type .llcrypt registered [ 7306.536874] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 7307.440463] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7307.748338] Lustre: Lustre: Build Version: 2.15.7_9_g1c128dc [ 7307.952954] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 7307.964108] LNet: Accept secure, port 988 [ 7309.632609] Key type lgssc registered [ 7310.581950] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7313.627497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7319.483176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7320.808299] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7320.879424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7324.698152] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 7326.188597] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7331.302891] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7332.516421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7332.601802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7332.704995] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7332.726897] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 7332.730539] Lustre: Permanently deactivating lustre-MDT0001 [ 7333.001585] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7336.151733] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7336.542954] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 7343.078868] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7347.461413] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7353.550734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7353.923199] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7357.411729] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 7362.114622] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7362.129603] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:163 to 0x280000400:193 [ 7363.601147] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7367.152527] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7369.561411] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7369.739489] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7371.842825] Lustre: lustre-OST0001: deleting orphan objects from 0x0:67 to 0x0:129 [ 7371.844656] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:193 [ 7374.035558] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 7381.289565] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7383.896312] Lustre: Permanently reactivating lustre-MDT0001 [ 7383.918556] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active in log lustre-client [ 7383.921299] Lustre: Skipped 3 previous similar messages [ 7387.005857] Lustre: lustre-MDT0001: Client ab029e3d-2abc-4819-9ef0-2c9908d600df (at 192.168.202.56@tcp) reconnecting [ 7391.876374] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7392.316530] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7392.328756] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7392.337267] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7392.350938] LustreError: 255195:0:(llog_osd.c:2141:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7392.353913] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to (at 0@lo) [ 7392.359502] LustreError: 255195:0:(lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7392.359638] LustreError: 255195:0:(obd_config.c:1999:class_config_llog_handler()) MGC192.168.202.156@tcp: cfg command failed: rc = -108 [ 7392.359661] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7392.359661] [ 7392.360575] LustreError: 251392:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7393.429235] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7397.608544] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 40 [ 7397.845917] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7401.376900] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 40 [ 7401.562979] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7403.693484] Lustre: Permanently deactivating lustre-MDT0001 [ 7403.714906] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active in log lustre-client [ 7403.727875] Lustre: Skipped 2 previous similar messages [ 7410.206879] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7423.973453] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7423.982086] Lustre: Skipped 2 previous similar messages [ 7423.992160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7423.996137] Lustre: Skipped 3 previous similar messages [ 7429.093552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7429.116449] Lustre: Skipped 3 previous similar messages [ 7429.565798] Lustre: server umount lustre-MDT0000 complete [ 7432.837449] LustreError: 251381:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761317783 with bad export cookie 13860271151238979190 [ 7432.840120] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7432.852789] LustreError: 251381:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7433.115572] Lustre: server umount lustre-MDT0001 complete [ 7442.898353] Lustre: server umount lustre-OST0000 complete [ 7454.398281] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 10:56:44 (1761317804) [ 7463.519849] Lustre: server umount lustre-OST0001 complete [ 7468.915970] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 7476.042346] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 7485.293109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7489.500356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7493.760583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7498.449855] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7503.377900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7503.420583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7503.596760] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 7503.599542] Lustre: Skipped 2 previous similar messages [ 7503.637374] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 7503.699416] Lustre: lustre-MDT0000: new disk, initializing [ 7503.779279] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7503.804030] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7507.070433] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 7515.155107] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7515.211401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7515.270541] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 7515.305896] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 7515.310065] Lustre: Skipped 1 previous similar message [ 7515.395834] Lustre: lustre-MDT0001: new disk, initializing [ 7515.488583] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7515.524622] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7515.531786] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7518.665045] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 7526.518287] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7531.928594] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7537.783922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7537.835048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7537.838886] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 7538.023404] Lustre: lustre-OST0000: new disk, initializing [ 7538.025728] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 7538.092726] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7541.595333] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 7544.885407] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7544.896309] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7548.890578] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7550.577195] LustreError: 259907:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7558.064316] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7558.139729] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7558.144288] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 7558.258390] Lustre: lustre-OST0001: new disk, initializing [ 7558.260961] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 7560.659239] LustreError: 259907:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 awake [ 7560.676940] LustreError: 259907:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7561.374919] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 7568.321336] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7568.436105] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7568.445398] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7569.372106] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7570.721778] LustreError: 259907:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 awake [ 7573.479922] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7573.487125] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7573.490191] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7573.498686] Lustre: Skipped 2 previous similar messages [ 7573.505984] Lustre: Skipped 1 previous similar message [ 7574.203417] Lustre: server umount lustre-OST0001 complete [ 7578.597494] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7578.603679] LustreError: Skipped 1 previous similar message [ 7588.836169] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7588.845666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7588.849352] Lustre: Skipped 1 previous similar message [ 7593.953818] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7593.962048] Lustre: Skipped 1 previous similar message [ 7599.080652] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7599.088411] LustreError: Skipped 7 previous similar messages [ 7602.145683] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 7602.228539] Lustre: server umount lustre-OST0000 complete [ 7607.777276] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7607.779920] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7607.794568] Lustre: Skipped 1 previous similar message [ 7607.797207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7607.800042] Lustre: Skipped 2 previous similar messages [ 7613.820304] LustreError: 260752:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761317964 with bad export cookie 13860271151238981752 [ 7613.822148] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7613.826814] LustreError: 260752:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7628.256287] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 7628.478361] Lustre: server umount lustre-MDT0001 complete [ 7628.481711] Lustre: Skipped 1 previous similar message [ 7636.043666] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 7637.673352] Key type lgssc unregistered [ 7637.872343] LNet: 265097:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7637.879683] LNet: Removed LNI 192.168.202.156@tcp [ 7638.387901] Key type .llcrypt unregistered [ 7638.389614] Key type ._llcrypt unregistered [ 7660.951402] alg: No test for adler32 (adler32-zlib) [ 7661.705434] Key type ._llcrypt registered [ 7661.707723] Key type .llcrypt registered [ 7661.793709] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 7671.341194] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 7672.187589] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7672.438681] Lustre: Lustre: Build Version: 2.15.7_9_g1c128dc [ 7672.596187] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 7672.599623] LNet: Accept secure, port 988 [ 7674.232131] Key type lgssc registered [ 7674.958909] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7681.187973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7686.281775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7690.715802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7695.475876] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7700.298521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7700.354704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7701.524514] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 7701.550750] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 7701.598182] Lustre: lustre-MDT0000: new disk, initializing [ 7701.647835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7701.657680] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7704.163906] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 7712.743635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7712.831969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7712.944039] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 7712.988402] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 7712.991386] Lustre: Skipped 1 previous similar message [ 7713.109045] Lustre: lustre-MDT0001: new disk, initializing [ 7713.233566] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7713.276555] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7713.299837] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7716.807440] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 7725.014313] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7729.552348] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7734.693939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7734.756913] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7734.764078] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 7734.936960] Lustre: lustre-OST0000: new disk, initializing [ 7734.946258] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 7735.035547] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7738.207803] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 7742.525459] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7742.531579] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7745.814376] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7748.896740] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 7749.041874] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7752.150965] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 7752.329392] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7758.306976] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7758.307082] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7758.316445] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7758.320103] Lustre: Skipped 1 previous similar message [ 7759.680112] Lustre: server umount lustre-OST0000 complete [ 7763.429312] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7763.432398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7763.440873] Lustre: Skipped 2 previous similar messages [ 7768.444425] Lustre: server umount lustre-MDT0000 complete [ 7768.550470] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7768.562683] LustreError: Skipped 1 previous similar message [ 7771.834022] LustreError: 269187:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761318122 with bad export cookie 11286178129724190647 [ 7771.834646] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7771.846394] LustreError: 269187:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7772.169215] Lustre: server umount lustre-MDT0001 complete [ 7778.960818] Lustre: DEBUG MARKER: == conf-sanity test 52: check recovering objects from lost+found ========================================================== 11:02:09 (1761318129) [ 7784.469916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7784.796288] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7784.854239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7787.804901] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 7790.049503] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7793.273474] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7793.589207] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7796.808278] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 7801.364341] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7805.213343] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7809.893342] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7810.089332] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7812.948596] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 7818.027678] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7833.570414] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7833.574738] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7833.585123] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7833.589925] LustreError: Skipped 1 previous similar message [ 7833.593870] Lustre: Skipped 1 previous similar message [ 7838.693189] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7838.700287] Lustre: Skipped 1 previous similar message [ 7838.830380] Lustre: server umount lustre-OST0000 complete [ 7841.975415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7849.942319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7850.164872] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7853.830619] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 7859.373335] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7859.390708] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7859.396564] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.202.156@tcp (at 0@lo) [ 7859.399473] Lustre: Skipped 1 previous similar message [ 7859.403926] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:65 [ 7859.507726] Lustre: lustre-OST0000: deleting orphan objects from 0x0:42 to 0x0:65 [ 7859.803722] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7869.925194] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7869.926414] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7869.931132] LustreError: Skipped 1 previous similar message [ 7869.931343] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7869.964735] Lustre: Skipped 2 previous similar messages [ 7873.365927] Lustre: server umount lustre-OST0000 complete [ 7880.161985] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7880.163485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7880.176454] Lustre: Skipped 1 previous similar message [ 7880.179261] Lustre: Skipped 1 previous similar message [ 7882.479940] Lustre: server umount lustre-MDT0000 complete [ 7885.282640] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7885.297388] LustreError: Skipped 1 previous similar message [ 7885.752759] LustreError: 276367:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761318236 with bad export cookie 11286178129724191501 [ 7885.759605] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7885.771870] LustreError: 276367:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7897.665637] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 7899.965376] Key type lgssc unregistered [ 7900.271583] LNet: 279188:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7900.282669] LNet: Removed LNI 192.168.202.156@tcp [ 7901.261608] Key type .llcrypt unregistered [ 7901.268193] Key type ._llcrypt unregistered [ 7917.964504] alg: No test for adler32 (adler32-zlib) [ 7918.716493] Key type ._llcrypt registered [ 7918.718885] Key type .llcrypt registered [ 7918.791619] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 7919.325346] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7919.620223] Lustre: Lustre: Build Version: 2.15.7_9_g1c128dc [ 7919.746076] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 7919.751064] LNet: Accept secure, port 988 [ 7921.360162] Key type lgssc registered [ 7922.093113] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7926.779493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7928.012320] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7928.048455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7930.767434] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 7933.417724] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7936.892677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7937.127903] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7940.049356] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 7945.019545] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7949.451535] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7954.806602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7955.027365] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7958.132568] Lustre: lustre-OST0000: deleting orphan objects from 0x0:42 to 0x0:97 [ 7958.224372] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 7963.598586] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7983.076451] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7983.081975] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7983.095507] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7983.095507] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7983.838078] Lustre: server umount lustre-OST0000 complete [ 7988.193883] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7988.194725] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7988.195378] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7988.200021] Lustre: Skipped 2 previous similar messages [ 7992.344948] Lustre: server umount lustre-MDT0000 complete [ 7993.313117] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7993.321669] LustreError: Skipped 1 previous similar message [ 7994.792571] LustreError: 280632:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761318345 with bad export cookie 5478443181743501639 [ 7994.794794] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7994.797346] LustreError: 280632:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7994.973706] Lustre: server umount lustre-MDT0001 complete [ 8002.968173] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 8004.927959] Key type lgssc unregistered [ 8005.127340] LNet: 285079:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8005.136886] LNet: Removed LNI 192.168.202.156@tcp [ 8005.681248] Key type .llcrypt unregistered [ 8005.682984] Key type ._llcrypt unregistered [ 8015.714206] alg: No test for adler32 (adler32-zlib) [ 8016.476413] Key type ._llcrypt registered [ 8016.478279] Key type .llcrypt registered [ 8016.569491] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 8017.497230] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8017.815208] Lustre: Lustre: Build Version: 2.15.7_9_g1c128dc [ 8017.984985] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 8017.990402] LNet: Accept secure, port 988 [ 8019.616536] Key type lgssc registered [ 8020.407181] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8026.034676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8027.375093] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8027.438426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8030.583591] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 8032.738789] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8036.354748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8036.626482] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8039.654786] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 8044.541822] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8048.315402] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8053.287194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8053.626584] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8056.553466] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 8057.710297] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 8062.131402] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8073.185939] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8073.189821] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8073.201564] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8077.282874] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8077.283668] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8077.284058] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8077.288477] Lustre: Skipped 1 previous similar message [ 8079.251945] Lustre: server umount lustre-OST0000 complete [ 8082.401925] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8082.403512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8082.410553] Lustre: Skipped 1 previous similar message [ 8082.416978] Lustre: Skipped 1 previous similar message [ 8087.521868] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8087.525906] Lustre: Skipped 1 previous similar message [ 8092.643890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8092.646916] Lustre: Skipped 1 previous similar message [ 8096.224571] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 8096.307811] Lustre: server umount lustre-MDT0000 complete [ 8098.793685] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8098.803595] LustreError: Skipped 1 previous similar message [ 8098.904181] LustreError: 286326:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761318449 with bad export cookie 9217662446038979322 [ 8098.905929] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8098.908981] LustreError: 286326:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8099.108330] Lustre: server umount lustre-MDT0001 complete [ 8107.547114] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 8109.649402] Key type lgssc unregistered [ 8109.858723] LNet: 290170:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8109.866614] LNet: Removed LNI 192.168.202.156@tcp [ 8110.527201] Key type .llcrypt unregistered [ 8110.528939] Key type ._llcrypt unregistered [ 8120.836202] alg: No test for adler32 (adler32-zlib) [ 8121.597568] Key type ._llcrypt registered [ 8121.599028] Key type .llcrypt registered [ 8121.696236] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 8122.327975] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8122.810604] Lustre: Lustre: Build Version: 2.15.7_9_g1c128dc [ 8123.025261] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 8123.028963] LNet: Accept secure, port 988 [ 8124.680417] Key type lgssc registered [ 8125.567969] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8130.708473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8131.936377] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8131.983544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8134.680454] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 8137.186727] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8139.818092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8140.060098] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8142.627859] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 8147.912246] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8151.847707] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8156.138247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8156.362365] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8158.901862] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 8162.545724] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 8163.669249] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8178.145683] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8178.152096] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8178.161084] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8181.219293] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8181.220962] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8181.222575] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8181.236602] Lustre: Skipped 1 previous similar message [ 8184.079804] Lustre: server umount lustre-OST0000 complete [ 8191.457609] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8191.462889] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8191.466607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8191.471762] Lustre: Skipped 1 previous similar message [ 8191.483588] Lustre: Skipped 1 previous similar message [ 8192.559236] Lustre: server umount lustre-MDT0000 complete [ 8194.839138] LustreError: 291417:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761318545 with bad export cookie 4874630444436297874 [ 8194.840616] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8194.857259] LustreError: 291417:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8195.079578] Lustre: server umount lustre-MDT0001 complete [ 8202.568755] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 8204.345304] Key type lgssc unregistered [ 8204.559134] LNet: 295109:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8204.564833] LNet: Removed LNI 192.168.202.156@tcp [ 8205.103207] Key type .llcrypt unregistered [ 8205.104993] Key type ._llcrypt unregistered [ 8220.944124] alg: No test for adler32 (adler32-zlib) [ 8221.697471] Key type ._llcrypt registered [ 8221.699031] Key type .llcrypt registered [ 8221.774769] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 8222.442349] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8222.846301] Lustre: Lustre: Build Version: 2.15.7_9_g1c128dc [ 8223.014275] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 8223.018489] LNet: Accept secure, port 988 [ 8224.672408] Key type lgssc registered [ 8225.487261] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8230.368955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8231.619830] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8231.665955] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8233.888333] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 8237.026160] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8238.537721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8238.717138] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8241.064128] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 8246.048242] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8250.270786] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8255.186799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8255.395232] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8258.235425] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 8262.574888] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:193 [ 8263.502761] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8283.115027] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8283.123867] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8283.133358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8284.130732] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8284.135443] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8284.146310] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8288.710252] Lustre: server umount lustre-OST0000 complete [ 8294.371184] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8294.373869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8294.377900] Lustre: Skipped 1 previous similar message [ 8294.390467] Lustre: Skipped 2 previous similar messages [ 8297.707945] Lustre: server umount lustre-MDT0000 complete [ 8299.491129] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8299.496096] LustreError: Skipped 1 previous similar message [ 8300.230745] LustreError: 296550:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761318651 with bad export cookie 9431867791083334009 [ 8300.242562] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8300.477497] Lustre: server umount lustre-MDT0001 complete [ 8307.703511] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 8309.400173] Key type lgssc unregistered [ 8309.585896] LNet: 301043:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8309.590456] LNet: Removed LNI 192.168.202.156@tcp [ 8310.079990] Key type .llcrypt unregistered [ 8310.082650] Key type ._llcrypt unregistered [ 8319.963168] alg: No test for adler32 (adler32-zlib) [ 8320.715473] Key type ._llcrypt registered [ 8320.717019] Key type .llcrypt registered [ 8320.780591] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 8321.382627] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8321.694529] Lustre: Lustre: Build Version: 2.15.7_9_g1c128dc [ 8321.832192] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 8321.837083] LNet: Accept secure, port 988 [ 8323.456134] Key type lgssc registered [ 8324.347764] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8330.071364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8331.400067] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8331.439711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8334.408487] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 8336.865748] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8339.568904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8339.781843] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8342.226525] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 8346.879987] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8350.802806] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8355.510695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8355.765374] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8358.450213] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 8361.895406] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:225 [ 8362.666992] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8374.242524] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8374.243022] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8374.247947] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8374.251634] Lustre: Skipped 1 previous similar message [ 8378.427781] Lustre: server umount lustre-OST0000 complete [ 8380.902802] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8380.914692] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8380.926717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8380.933518] Lustre: Skipped 1 previous similar message [ 8384.481448] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8384.482554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8386.661989] Lustre: server umount lustre-MDT0000 complete [ 8388.903902] LustreError: 305340:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761318739 with bad export cookie 10701192719923436894 [ 8388.904704] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8388.910392] LustreError: 305340:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8389.090736] Lustre: server umount lustre-MDT0001 complete [ 8396.065217] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 8397.754671] Key type lgssc unregistered [ 8397.952583] LNet: 306143:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8397.960532] LNet: Removed LNI 192.168.202.156@tcp [ 8398.539140] Key type .llcrypt unregistered [ 8398.541380] Key type ._llcrypt unregistered [ 8406.846848] alg: No test for adler32 (adler32-zlib) [ 8407.600437] Key type ._llcrypt registered [ 8407.602097] Key type .llcrypt registered [ 8407.653191] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 8408.166127] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8408.360797] Lustre: Lustre: Build Version: 2.15.7_9_g1c128dc [ 8408.462489] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 8408.465508] LNet: Accept secure, port 988 [ 8410.072137] Key type lgssc registered [ 8410.638760] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8415.446699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8416.687132] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8416.733181] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8419.166618] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 8421.859578] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8424.367462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8424.581883] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8426.910783] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 8431.497718] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8435.183705] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8439.304946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8439.510244] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8441.875914] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 8445.713878] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8446.695235] Lustre: lustre-OST0000: deleting orphan objects from 0x0:227 to 0x0:257 [ 8456.160994] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8456.164830] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8456.170972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8457.188085] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8457.191808] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8457.196974] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8460.657640] Lustre: server umount lustre-OST0000 complete [ 8465.377885] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8465.384301] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8465.391240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8465.395754] Lustre: Skipped 1 previous similar message [ 8467.425123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8467.425618] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8472.546311] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8472.549322] Lustre: Skipped 2 previous similar messages [ 8477.152147] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 8477.210321] Lustre: server umount lustre-MDT0000 complete [ 8477.666855] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8477.681315] LustreError: Skipped 1 previous similar message [ 8479.112414] LustreError: 308194:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761318830 with bad export cookie 5937523654840182962 [ 8479.113929] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8479.117627] LustreError: 308194:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8479.275233] Lustre: server umount lustre-MDT0001 complete [ 8485.445667] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 8486.958437] Key type lgssc unregistered [ 8487.100393] LNet: 311063:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8487.104744] LNet: Removed LNI 192.168.202.156@tcp [ 8487.550678] Key type .llcrypt unregistered [ 8487.552382] Key type ._llcrypt unregistered [ 8494.553471] alg: No test for adler32 (adler32-zlib) [ 8495.311073] Key type ._llcrypt registered [ 8495.312764] Key type .llcrypt registered [ 8495.368622] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8504.416340] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 8511.597152] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 8512.059462] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8512.348340] Lustre: Lustre: Build Version: 2.15.7_9_g1c128dc [ 8512.492040] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 8512.495740] LNet: Accept secure, port 988 [ 8514.121071] Key type lgssc registered [ 8514.721963] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8519.819202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8523.717703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8527.091707] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8530.515904] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8534.106355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8534.145421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8535.289912] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 8535.311700] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 8535.358887] Lustre: lustre-MDT0000: new disk, initializing [ 8535.421636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8535.442225] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8537.928240] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 8544.353439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8544.392551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8544.451373] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 8544.488926] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 8544.493452] Lustre: Skipped 1 previous similar message [ 8544.547696] Lustre: lustre-MDT0001: new disk, initializing [ 8544.615720] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8544.642759] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8544.656371] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8547.200578] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 8552.868801] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8556.315585] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8560.548973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8560.598283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8560.603399] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 8560.749679] Lustre: lustre-OST0000: new disk, initializing [ 8560.752810] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 8560.813908] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8563.268468] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 8566.298636] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8566.304926] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8568.526374] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8571.093137] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 8571.255951] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8573.459505] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 8573.575259] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8576.483267] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8576.483322] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8576.484078] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8576.486968] LustreError: Skipped 1 previous similar message [ 8576.492665] Lustre: Skipped 1 previous similar message [ 8580.784452] Lustre: server umount lustre-OST0000 complete [ 8585.699948] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8585.703354] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8585.711089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8585.713491] Lustre: Skipped 1 previous similar message [ 8586.723130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8586.726141] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8589.293958] Lustre: server umount lustre-MDT0000 complete [ 8591.593820] LustreError: 314798:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761318942 with bad export cookie 5439607791083996413 [ 8591.596697] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8591.603912] LustreError: 314798:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8591.829824] Lustre: server umount lustre-MDT0001 complete [ 8596.516794] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 11:15:46 (1761318946) [ 8600.339187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8600.577694] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8600.620642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8603.180102] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 8605.670464] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8608.221725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8608.442156] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8610.682155] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 8614.257413] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8617.197139] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8620.586535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8620.761678] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8622.780107] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 8626.227924] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8636.385294] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8636.386941] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8636.394875] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8636.397415] Lustre: Skipped 1 previous similar message [ 8636.401678] Lustre: Skipped 1 previous similar message [ 8641.507814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8641.510832] Lustre: Skipped 1 previous similar message [ 8641.847381] Lustre: server umount lustre-OST0000 complete [ 8644.578425] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8644.582241] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8651.750622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8651.753509] Lustre: Skipped 3 previous similar messages [ 8658.400130] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 8658.473671] Lustre: server umount lustre-MDT0000 complete [ 8660.730824] LustreError: 320137:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761319011 with bad export cookie 5439607791083997267 [ 8660.732809] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8660.740603] LustreError: 320137:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8660.749982] LustreError: 320140:0:(osp_object.c:629:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 8660.755634] LustreError: 320140:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) lustre-MDT0000-osp-MDT0001: fail to cancel 1 llog-records: rc = -5 [ 8660.760539] LustreError: 320140:0:(llog_cat.c:789:llog_cat_cancel_records()) lustre-MDT0000-osp-MDT0001: fail to cancel 1 of 1 llog-records: rc = -5 [ 8668.199798] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 8669.524740] Key type lgssc unregistered [ 8669.663950] LNet: 323007:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8669.671204] LNet: Removed LNI 192.168.202.156@tcp [ 8670.065319] Key type .llcrypt unregistered [ 8670.066888] Key type ._llcrypt unregistered [ 8684.806951] alg: No test for adler32 (adler32-zlib) [ 8685.558341] Key type ._llcrypt registered [ 8685.559824] Key type .llcrypt registered [ 8685.617321] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 8692.783690] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 8693.392790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8693.601517] Lustre: Lustre: Build Version: 2.15.7_9_g1c128dc [ 8693.722868] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 8693.724957] LNet: Accept secure, port 988 [ 8695.336131] Key type lgssc registered [ 8695.817202] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8700.969139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8704.835105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8708.068930] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8711.113950] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8713.475155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8715.765487] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8719.622147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8719.691490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8720.846231] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 8720.865398] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 8720.908475] Lustre: lustre-MDT0000: new disk, initializing [ 8720.975846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8720.990733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8723.351668] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 8729.333861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8729.381898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8729.447879] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 8729.477827] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 8729.482564] Lustre: Skipped 1 previous similar message [ 8729.530268] Lustre: lustre-MDT0001: new disk, initializing [ 8729.595784] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8729.620209] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8729.624586] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8732.274791] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 8738.129370] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8741.092725] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8744.293262] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8744.339473] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8744.343622] Lustre: lustre-OST2710-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 8744.449429] Lustre: lustre-OST2710: new disk, initializing [ 8744.452343] Lustre: srv-lustre-OST2710: No data found on store. Initialize space: rc = -61 [ 8744.488973] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8746.422307] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 8751.472166] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 8752.647444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8752.653719] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8754.473519] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8754.506877] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8754.510876] Lustre: lustre-OST03e8-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 8754.571974] Lustre: lustre-OST03e8: new disk, initializing [ 8754.574874] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space: rc = -61 [ 8754.610062] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8756.548868] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 8759.773975] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 8760.852440] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8760.858818] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8763.296725] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 40 [ 8763.433162] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8765.680121] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 40 [ 8765.790508] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8768.130599] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 40 [ 8768.263883] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8770.320210] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 40 [ 8770.422940] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8775.650235] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8775.653174] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8775.660290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8776.164305] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8776.164690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8776.173474] Lustre: Skipped 2 previous similar messages [ 8776.181256] Lustre: Skipped 3 previous similar messages [ 8778.075757] Lustre: server umount lustre-MDT0000 complete [ 8780.192449] LustreError: 326945:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761319131 with bad export cookie 9107945342900538232 [ 8780.198826] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8780.210262] LustreError: 326945:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8780.414628] Lustre: server umount lustre-MDT0001 complete [ 8788.384180] Lustre: 324748:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761319132/real 1761319132] req@00000000e0170e1f x1846876877906304/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761319139 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 8788.397104] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8788.960489] Lustre: 332434:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761319133/real 1761319133] req@0000000053f12c92 x1846876877906432/t0(0) o39->lustre-MDT0001-lwp-OST2710@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761319139 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 8789.020537] Lustre: server umount lustre-OST2710 complete [ 8791.358474] Lustre: server umount lustre-OST03e8 complete [ 8800.103891] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 8804.501629] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 8808.921351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8811.848584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8814.843836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8818.024671] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8821.261491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8821.300548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8821.414988] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 8821.433265] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 8821.477903] Lustre: lustre-MDT0000: new disk, initializing [ 8821.523898] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8821.533570] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8823.460810] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 8828.783376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8828.814098] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8828.859336] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 8828.979069] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8830.846359] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 8835.870514] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8838.326845] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8841.605587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8841.642922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8841.646990] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 8841.774395] Lustre: lustre-OST0000: new disk, initializing [ 8841.776414] Lustre: Skipped 1 previous similar message [ 8841.778995] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 8841.781878] Lustre: Skipped 2 previous similar messages [ 8841.819747] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8841.822792] Lustre: Skipped 1 previous similar message [ 8843.776571] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 8845.767631] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8845.772616] Lustre: Skipped 1 previous similar message [ 8845.774902] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8848.726722] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8850.661537] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 8850.758546] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8852.664569] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 8852.771752] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8856.034504] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8856.035118] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8856.036119] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8856.050565] Lustre: Skipped 1 previous similar message [ 8859.742061] Lustre: server umount lustre-OST0000 complete [ 8864.737862] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8864.741769] LustreError: Skipped 1 previous similar message [ 8864.743981] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8864.749967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8864.752844] Lustre: Skipped 1 previous similar message [ 8869.518218] LustreError: 336398:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761319220 with bad export cookie 9107945342900539856 [ 8869.521406] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8869.525151] LustreError: 336398:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8869.665815] Lustre: server umount lustre-MDT0001 complete [ 8869.667846] Lustre: Skipped 1 previous similar message [ 8873.201921] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 11:20:23 (1761319223) [ 8873.872529] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8874.667030] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 11:20:25 (1761319225) [ 8876.097418] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 8879.305135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8880.818101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8882.483742] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8884.046281] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8884.526645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8887.809409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8887.942613] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8887.958730] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 8888.058748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8890.049300] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 8894.294784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8894.353689] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8896.497374] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 8900.008434] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8902.628795] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8905.762032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8905.797489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8905.886556] Lustre: Denying initial registration attempt from nid 192.168.202.156@tcp, specified as failover [ 8905.891943] LustreError: 160-7: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8905.896522] LustreError: 343558:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -99 [ 8905.900428] LustreError: 343558:0:(obd_mount_server.c:1644:server_put_super()) no obd lustre-OST0000 [ 8905.903242] LustreError: 343558:0:(obd_mount_server.c:131:server_deregister_mount()) lustre-OST0000 not registered [ 8905.960664] Lustre: server umount lustre-OST0000 complete [ 8905.962967] LustreError: 343558:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8909.793246] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8909.797466] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8909.805336] Lustre: Skipped 1 previous similar message [ 8909.808117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8909.810460] Lustre: Skipped 2 previous similar messages [ 8916.071069] LustreError: 341641:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761319267 with bad export cookie 9107945342900540710 [ 8916.073388] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8916.076099] LustreError: 341641:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8922.000422] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 8923.312907] Key type lgssc unregistered [ 8923.464984] LNet: 344630:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8923.468838] LNet: Removed LNI 192.168.202.156@tcp [ 8923.830082] Key type .llcrypt unregistered [ 8923.831675] Key type ._llcrypt unregistered [ 8929.829220] alg: No test for adler32 (adler32-zlib) [ 8930.582362] Key type ._llcrypt registered [ 8930.584598] Key type .llcrypt registered [ 8930.643777] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 8931.173266] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8931.363159] Lustre: Lustre: Build Version: 2.15.7_9_g1c128dc [ 8931.485910] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 8931.490697] LNet: Accept secure, port 988 [ 8933.096155] Key type lgssc registered [ 8933.698909] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8937.100175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8938.810705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8940.594771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8942.549989] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8943.154035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8951.006552] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 8955.635570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8956.792053] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8956.806250] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 8956.909691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8959.139767] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 8963.382728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8963.455614] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8963.479615] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 8963.613879] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8965.868074] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 8969.495421] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8972.325875] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8975.704795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8975.747032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8975.845171] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8975.956286] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8978.270362] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 8982.729391] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8988.131936] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8988.132188] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8988.133513] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8988.135890] LustreError: Skipped 1 previous similar message [ 8988.141916] Lustre: Skipped 1 previous similar message [ 8993.250034] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8993.252836] Lustre: Skipped 2 previous similar messages [ 8998.368765] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 8998.371549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8998.376417] Lustre: Skipped 1 previous similar message [ 8998.462236] Lustre: server umount lustre-OST0000 complete [ 9003.492190] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9003.493575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9003.499114] Lustre: Skipped 1 previous similar message [ 9006.356728] Lustre: server umount lustre-MDT0000 complete [ 9007.994140] LustreError: 347497:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761319358 with bad export cookie 13765833725241087739 [ 9007.996899] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9007.998927] LustreError: 347497:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9008.127975] Lustre: server umount lustre-MDT0001 complete [ 9012.491893] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 9013.643293] Key type lgssc unregistered [ 9013.786572] LNet: 351196:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9013.790607] LNet: Removed LNI 192.168.202.156@tcp [ 9014.135596] Key type .llcrypt unregistered [ 9014.138188] Key type ._llcrypt unregistered [ 9025.116646] alg: No test for adler32 (adler32-zlib) [ 9025.868543] Key type ._llcrypt registered [ 9025.870175] Key type .llcrypt registered [ 9025.924199] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 9026.309176] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9026.577350] Lustre: Lustre: Build Version: 2.15.7_9_g1c128dc [ 9026.679669] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 9026.682104] LNet: Accept secure, port 988 [ 9028.280148] Key type lgssc registered [ 9028.752903] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9032.140063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9033.318201] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9033.344847] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9035.095413] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 9038.672163] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9038.830495] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9040.676732] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 9043.713709] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9045.994313] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9048.818477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9048.958790] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9050.742221] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 9053.891802] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9067.513669] LustreError: 355324:0:(ldlm_resource.c:1127:ldlm_resource_complain()) mdt-lustre-MDT0000_UUID: namespace resource [0x200001b71:0xf:0x0].0x0 (000000004ee70c4d) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 9068.514912] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9068.516191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9068.520406] Lustre: Skipped 1 previous similar message [ 9069.683175] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.56@tcp (stopping) [ 9069.685875] Lustre: Skipped 2 previous similar messages [ 9073.618725] Lustre: server umount lustre-MDT0000 complete [ 9073.635812] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9073.641633] LustreError: Skipped 2 previous similar messages [ 9074.793626] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9075.257492] LustreError: 352635:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761319426 with bad export cookie 12025972495946034918 [ 9075.261528] LustreError: 352635:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9075.262041] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9078.753352] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9078.753522] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9078.759043] Lustre: Skipped 1 previous similar message [ 9078.767844] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9080.801182] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9080.804329] Lustre: Skipped 1 previous similar message [ 9081.474239] Lustre: server umount lustre-MDT0001 complete [ 9084.205550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9087.842207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9092.066450] Lustre: Evicted from MGS (at 192.168.202.156@tcp) after server handle changed from 0xa6e4d644cb7b2ee6 to 0xa6e4d644cb7b4ac3 [ 9092.073250] Lustre: MGC192.168.202.156@tcp: Connection restored to 192.168.202.156@tcp (at 0@lo) [ 9092.207347] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9092.215239] LustreError: Skipped 2 previous similar messages [ 9092.242688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9092.279332] Lustre: lustre-OST0000: deleting orphan objects from 0x0:166 to 0x0:193 [ 9094.347207] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 9098.411198] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9098.588849] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9100.846353] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 9103.846873] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 9104.392833] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9106.916709] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 9110.240363] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 5 sec [ 9112.740703] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9113.413470] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9117.155718] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9117.156544] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9117.161495] Lustre: Skipped 1 previous similar message [ 9121.330711] Lustre: server umount lustre-OST0000 complete [ 9124.321197] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9124.324323] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9127.393045] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9127.395775] Lustre: Skipped 3 previous similar messages [ 9129.033759] Lustre: server umount lustre-MDT0000 complete [ 9130.507954] LustreError: 355528:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761319481 with bad export cookie 12025972495946042051 [ 9130.510850] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9130.512558] LustreError: 355528:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9136.063165] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 9137.290593] Key type lgssc unregistered [ 9137.424519] LNet: 359167:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9137.428740] LNet: Removed LNI 192.168.202.156@tcp [ 9137.749570] Key type .llcrypt unregistered [ 9137.751240] Key type ._llcrypt unregistered [ 9149.109769] alg: No test for adler32 (adler32-zlib) [ 9149.861403] Key type ._llcrypt registered [ 9149.863032] Key type .llcrypt registered [ 9149.908727] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 9150.397805] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9150.562602] Lustre: Lustre: Build Version: 2.15.7_9_g1c128dc [ 9150.657435] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 9150.661269] LNet: Accept secure, port 988 [ 9152.256178] Key type lgssc registered [ 9152.769562] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9156.193055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9157.388364] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9157.415819] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9159.126397] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 9162.525575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9162.692907] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9164.388583] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 9167.550552] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9170.137553] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9171.409223] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9171.409807] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9171.415929] Lustre: Skipped 1 previous similar message [ 9171.420670] Lustre: Skipped 1 previous similar message [ 9176.545727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9176.549279] Lustre: Skipped 1 previous similar message [ 9177.553947] Lustre: server umount lustre-MDT0000 complete [ 9177.571485] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9177.592526] LustreError: Skipped 1 previous similar message [ 9179.420279] LustreError: 362355:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761319530 with bad export cookie 3450331930326183238 [ 9179.421273] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9179.429029] LustreError: 362355:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9179.578730] Lustre: server umount lustre-MDT0001 complete [ 9186.449070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9186.609412] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9186.645578] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 9186.749750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9188.898399] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 9192.800218] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9192.855312] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9192.872954] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 9192.994139] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9195.028616] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 9198.625716] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9201.227470] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9204.842788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9204.944546] LustreError: 13b-9: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9204.951168] LustreError: 160-7: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9204.955893] LustreError: 365318:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -2 [ 9204.959838] LustreError: 365318:0:(obd_mount_server.c:1644:server_put_super()) no obd lustre-OST0000 [ 9204.962560] LustreError: 365318:0:(obd_mount_server.c:131:server_deregister_mount()) lustre-OST0000 not registered [ 9205.018762] Lustre: server umount lustre-OST0000 complete [ 9205.020829] LustreError: 365318:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9208.054638] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9208.166584] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9208.282636] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9210.192903] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 9213.450970] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9216.034454] Lustre: lustre-OST0000: deleting orphan objects from 0x0:166 to 0x0:225 [ 9216.748419] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9216.783516] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9216.787717] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 9216.819046] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9216.861833] Lustre: lustre-OST0001: new disk, initializing [ 9216.864933] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 9216.908985] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9218.836898] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 9222.162597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9222.167821] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9222.248209] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 9227.234818] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9227.236398] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9227.240602] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9227.254714] Lustre: Skipped 1 previous similar message [ 9229.684388] Lustre: server umount lustre-OST0001 complete [ 9232.354329] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9232.355574] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9232.360286] LustreError: Skipped 1 previous similar message [ 9232.366326] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9232.370956] Lustre: Skipped 2 previous similar messages [ 9237.484706] Lustre: server umount lustre-OST0000 complete [ 9242.596192] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9242.600068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9242.601181] Lustre: Skipped 2 previous similar messages [ 9247.720050] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9247.749068] LustreError: Skipped 1 previous similar message [ 9249.052215] LustreError: 363314:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761319600 with bad export cookie 3450331930326183777 [ 9249.052215] LustreError: 364117:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761319600 with bad export cookie 3450331930326183777 [ 9249.052888] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9249.070607] LustreError: 364117:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9249.873874] Lustre: server umount lustre-MDT0001 complete [ 9249.875736] Lustre: Skipped 1 previous similar message [ 9265.730145] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 9270.008390] Key type lgssc unregistered [ 9270.430200] LNet: 368688:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9270.446846] LNet: Removed LNI 192.168.202.156@tcp [ 9271.653381] Key type .llcrypt unregistered [ 9271.658871] Key type ._llcrypt unregistered [ 9305.401752] alg: No test for adler32 (adler32-zlib) [ 9306.156591] Key type ._llcrypt registered [ 9306.158521] Key type .llcrypt registered [ 9306.246908] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 9318.550981] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 9319.536660] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9319.849759] Lustre: Lustre: Build Version: 2.15.7_9_g1c128dc [ 9320.026438] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 9320.030512] LNet: Accept secure, port 988 [ 9321.706216] Key type lgssc registered [ 9322.654516] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9329.345639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9333.790844] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9337.820069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9342.216402] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9346.811076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9346.860475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9348.062338] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 9348.109714] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 9348.218517] Lustre: lustre-MDT0000: new disk, initializing [ 9348.318127] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9348.329114] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9351.667893] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 9359.567527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9359.617542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9359.692066] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 9359.724292] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 9359.728570] Lustre: Skipped 1 previous similar message [ 9359.776417] Lustre: lustre-MDT0001: new disk, initializing [ 9359.836956] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9359.865057] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9359.875914] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9362.599847] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 9369.278841] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9373.777921] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9378.653789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9378.700774] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9378.704727] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 9378.848924] Lustre: lustre-OST0000: new disk, initializing [ 9378.853413] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 9378.915884] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9382.008362] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 9386.546707] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9386.555043] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9389.681954] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9392.585119] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 9392.745182] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9395.661899] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 9395.829893] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9397.220265] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9397.223425] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9397.225152] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9397.225638] Lustre: Skipped 1 previous similar message [ 9402.346791] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9402.351063] Lustre: Skipped 2 previous similar messages [ 9403.431135] Lustre: server umount lustre-OST0000 complete [ 9407.457900] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9407.459134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9407.463637] Lustre: Skipped 1 previous similar message [ 9412.347435] Lustre: server umount lustre-MDT0000 complete [ 9412.577598] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9412.585641] LustreError: Skipped 1 previous similar message [ 9414.976254] LustreError: 372778:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761319765 with bad export cookie 11448476252552721553 [ 9414.981496] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9415.186396] Lustre: server umount lustre-MDT0001 complete [ 9421.327774] Lustre: DEBUG MARKER: == conf-sanity test 60: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 11:29:31 (1761319771) [ 9426.032609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9431.197921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9451.376314] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 9459.429469] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 9468.273574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9473.184237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9478.454980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9483.913141] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9490.140610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9490.209941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9490.504940] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 9490.533073] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 9490.607424] Lustre: lustre-MDT0000: new disk, initializing [ 9490.693420] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9490.704303] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9494.318231] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 9503.666617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9503.733441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9503.795692] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 9503.842300] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 9503.850051] Lustre: Skipped 1 previous similar message [ 9503.916458] Lustre: lustre-MDT0001: new disk, initializing [ 9504.002702] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9504.041403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9504.055428] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9508.770739] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 9518.230202] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9523.736278] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9529.780618] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9529.841564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9529.846964] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 9530.044774] Lustre: lustre-OST0000: new disk, initializing [ 9530.052511] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 9530.149590] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9534.386880] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 9543.194311] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9546.639822] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 9546.850380] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9550.461736] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 9550.647239] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9556.970096] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9556.972176] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9556.976910] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9556.976918] Lustre: Skipped 1 previous similar message [ 9557.020834] Lustre: Skipped 1 previous similar message [ 9558.647621] Lustre: server umount lustre-OST0000 complete [ 9562.083521] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9562.098688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9562.105041] Lustre: Skipped 2 previous similar messages [ 9568.149620] Lustre: server umount lustre-MDT0000 complete [ 9571.391959] LustreError: 380544:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761319922 with bad export cookie 11448476252552722407 [ 9571.393554] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9571.401031] LustreError: 380544:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 9579.264183] Lustre: DEBUG MARKER: == conf-sanity test 61: large xattr ====================== 11:32:09 (1761319929) [ 9587.274620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9587.699931] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9587.790194] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9591.826902] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 9592.806379] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9597.922483] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9599.170238] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9603.224272] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 9610.611380] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9616.498714] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9622.655376] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9623.051558] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9623.059744] Lustre: Skipped 1 previous similar message [ 9626.873068] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 9629.169179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9629.186341] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9634.669176] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9646.152311] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61.conf-sanity [ 9647.672817] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61.conf-sanity [ 9649.178591] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61.conf-sanity [ 9650.596374] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61.conf-sanity after remounting MDS [ 9651.938568] Lustre: Failing over lustre-MDT0000 [ 9652.091504] Lustre: server umount lustre-MDT0000 complete [ 9652.092906] Lustre: Skipped 1 previous similar message [ 9652.705527] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9652.709716] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9652.711973] Lustre: Skipped 3 previous similar messages [ 9652.723430] LustreError: Skipped 2 previous similar messages [ 9659.874382] Lustre: 371010:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761320003/real 1761320003] req@00000000b346f78b x1846877535388032/t0(0) o400->MGC192.168.202.156@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761320010 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 9659.886504] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9661.038446] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9661.049051] LustreError: Skipped 7 previous similar messages [ 9669.840777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9677.283520] Lustre: Evicted from MGS (at 192.168.202.156@tcp) after server handle changed from 0x9ee1287ebce67b36 to 0x9ee1287ebce67f1f [ 9677.283704] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9677.301092] Lustre: MGC192.168.202.156@tcp: Connection restored to 192.168.202.156@tcp (at 0@lo) [ 9677.310183] LustreError: Skipped 18 previous similar messages [ 9677.548570] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9679.469915] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9680.697910] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 9682.921923] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 9682.934224] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 9682.952437] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 9686.622843] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9687.584611] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9690.091671] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61.conf-sanity [ 9691.170517] Lustre: Failing over lustre-MDT0000 [ 9691.313844] Lustre: server umount lustre-MDT0000 complete [ 9693.156209] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9693.156633] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9693.165680] Lustre: Skipped 1 previous similar message [ 9700.192437] Lustre: 371009:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761320044/real 1761320044] req@0000000061214897 x1846877535397504/t0(0) o400->MGC192.168.202.156@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761320051 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 9700.202051] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9707.055509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9711.586457] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9711.593025] LustreError: Skipped 22 previous similar messages [ 9717.735378] Lustre: Evicted from MGS (at 192.168.202.156@tcp) after server handle changed from 0x9ee1287ebce67f1f to 0x9ee1287ebce68252 [ 9717.749175] Lustre: MGC192.168.202.156@tcp: Connection restored to (at 0@lo) [ 9717.755765] Lustre: Skipped 2 previous similar messages [ 9717.910047] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9717.914549] Lustre: Skipped 1 previous similar message [ 9717.947646] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9718.392558] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9720.304683] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 9723.373517] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 9723.386421] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 9723.423704] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:97 [ 9727.294377] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9728.247377] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9733.602933] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9733.602976] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9733.604285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9733.604292] Lustre: Skipped 2 previous similar messages [ 9733.622853] Lustre: Skipped 2 previous similar messages [ 9737.792896] Lustre: server umount lustre-MDT0000 complete [ 9740.133890] LustreError: 385179:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761320091 with bad export cookie 11448476252552725074 [ 9740.139195] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9740.140602] LustreError: 385179:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9748.961726] Lustre: 391008:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761320093/real 1761320093] req@0000000019cf8ed1 x1846877535408320/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761320099 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 9758.133323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9761.127974] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 9767.095406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9770.431322] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 9776.255147] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9780.103137] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9784.811654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9788.082345] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:129 [ 9788.383395] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 9794.290448] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9802.725702] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9802.726030] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9802.745792] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9802.745802] Lustre: Skipped 2 previous similar messages [ 9802.762938] Lustre: Skipped 2 previous similar messages [ 9808.727158] Lustre: server umount lustre-OST0000 complete [ 9808.730046] Lustre: Skipped 2 previous similar messages [ 9818.083764] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9818.092352] LustreError: Skipped 13 previous similar messages [ 9820.463850] LustreError: 391815:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761320171 with bad export cookie 11448476252552725781 [ 9820.464959] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9820.470241] LustreError: 391815:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9829.905466] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 9832.006745] Key type lgssc unregistered [ 9832.194173] LNet: 395484:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9832.204525] LNet: Removed LNI 192.168.202.156@tcp [ 9832.807449] Key type .llcrypt unregistered [ 9832.809419] Key type ._llcrypt unregistered [ 9848.926893] alg: No test for adler32 (adler32-zlib) [ 9849.678448] Key type ._llcrypt registered [ 9849.679917] Key type .llcrypt registered [ 9849.767632] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 9850.421236] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9850.861979] Lustre: Lustre: Build Version: 2.15.7_9_g1c128dc [ 9851.024106] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 9851.028495] LNet: Accept secure, port 988 [ 9852.674229] Key type lgssc registered [ 9853.461039] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9859.143764] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9859.149939] LustreError: 396968:0:(osd_handler.c:8211:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 9859.154611] LustreError: 396968:0:(obd_config.c:774:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 9859.159656] LustreError: 396968:0:(obd_mount.c:200:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 9859.163508] LustreError: 396968:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 9859.167594] LustreError: 396968:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 9863.943354] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9863.948536] LustreError: 397366:0:(osd_handler.c:8211:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 9863.955425] LustreError: 397366:0:(obd_config.c:774:class_setup()) setup lustre-OST0000-osd failed (-22) [ 9863.961243] LustreError: 397366:0:(obd_mount.c:200:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 9863.965191] LustreError: 397366:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 9863.971559] LustreError: 397366:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 9875.430996] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 9877.467530] Key type lgssc unregistered [ 9877.692159] LNet: 398323:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9877.707286] LNet: Removed LNI 192.168.202.156@tcp [ 9878.326075] Key type .llcrypt unregistered [ 9878.329775] Key type ._llcrypt unregistered [ 9891.211191] alg: No test for adler32 (adler32-zlib) [ 9891.969717] Key type ._llcrypt registered [ 9891.971712] Key type .llcrypt registered [ 9892.039411] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 9901.961588] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 9902.675214] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9902.988391] Lustre: Lustre: Build Version: 2.15.7_9_g1c128dc [ 9903.150235] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 9903.154708] LNet: Accept secure, port 988 [ 9904.776140] Key type lgssc registered [ 9905.523812] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9911.828379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9915.993035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9919.600464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9922.820634] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9926.340348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9926.390777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9927.520501] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 9927.538768] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 9927.579145] Lustre: lustre-MDT0000: new disk, initializing [ 9927.626574] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9927.637947] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9929.828804] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 9936.298525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9936.329562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9936.381343] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 9936.407909] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 9936.411450] Lustre: Skipped 1 previous similar message [ 9936.458294] Lustre: lustre-MDT0001: new disk, initializing [ 9936.512712] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9936.536956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9936.544167] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9938.806063] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 9943.873209] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9946.766372] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9950.130973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9950.172861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9950.177114] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 9950.301187] Lustre: lustre-OST0000: new disk, initializing [ 9950.304153] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 9950.348619] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9952.527202] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [ 9957.386354] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9957.393535] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9957.437365] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9959.486522] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 9959.600480] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9961.906480] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 9962.026866] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9967.585169] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9967.587719] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9967.591266] LustreError: Skipped 1 previous similar message [ 9967.599174] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9967.606566] Lustre: Skipped 1 previous similar message [ 9972.706663] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9972.709530] Lustre: Skipped 1 previous similar message [ 9977.312137] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 9977.436667] Lustre: server umount lustre-OST0000 complete [ 9982.947323] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9982.947936] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9982.954229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9982.957048] Lustre: Skipped 2 previous similar messages [ 9982.962615] Lustre: Skipped 1 previous similar message [ 9985.699851] Lustre: server umount lustre-MDT0000 complete [ 9987.769093] LustreError: 401589:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761320338 with bad export cookie 10516008655475918418 [ 9987.775642] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9987.775933] LustreError: 401589:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9987.953717] Lustre: server umount lustre-MDT0001 complete [ 9993.175303] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 9994.240315] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 11:39:04 (1761320344) [ 9998.019575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9998.250793] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9998.289980] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10000.565937] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [10003.426989] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10005.273815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10005.498499] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10008.080738] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [10012.975651] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10016.725268] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10020.417053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10020.584192] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10022.937920] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [10026.938346] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10030.766264] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10030.831787] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10030.840657] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [10030.925556] Lustre: lustre-OST0001: new disk, initializing [10030.929146] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [10033.642816] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [10038.969923] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [10045.564499] Lustre: lustre-OST0001: Not available for connect from 192.168.202.56@tcp (stopping) [10045.932282] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10045.935943] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10045.944924] Lustre: Skipped 1 previous similar message [10047.362828] Lustre: server umount lustre-OST0001 complete [10051.045658] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10051.054068] LustreError: Skipped 1 previous similar message [10056.164299] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10056.170491] LustreError: Skipped 1 previous similar message [10061.283358] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10061.283917] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10061.285469] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10061.285714] Lustre: Skipped 2 previous similar messages [10061.289768] LustreError: Skipped 1 previous similar message [10061.294939] Lustre: Skipped 1 previous similar message [10067.133549] Lustre: server umount lustre-OST0000 complete [10071.521787] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10071.534823] Lustre: Skipped 1 previous similar message [10071.534993] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10071.540197] Lustre: Skipped 4 previous similar messages [10075.570836] Lustre: server umount lustre-MDT0000 complete [10076.641888] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10076.647760] LustreError: Skipped 3 previous similar messages [10077.622764] LustreError: 406177:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761320428 with bad export cookie 10516008655475919272 [10077.623926] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10077.630278] LustreError: 406177:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [10091.378832] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [10093.137699] Key type lgssc unregistered [10093.381351] LNet: 411094:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10093.391969] LNet: Removed LNI 192.168.202.156@tcp [10093.917980] Key type .llcrypt unregistered [10093.919767] Key type ._llcrypt unregistered [10114.812250] alg: No test for adler32 (adler32-zlib) [10115.564440] Key type ._llcrypt registered [10115.566730] Key type .llcrypt registered [10115.634560] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [10124.401107] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [10125.040608] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10125.273419] Lustre: Lustre: Build Version: 2.15.7_9_g1c128dc [10125.416518] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [10125.422134] LNet: Accept secure, port 988 [10127.048137] Key type lgssc registered [10127.647122] Lustre: Echo OBD driver; http://www.lustre.org/ [10133.030931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10137.789175] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10141.398869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10145.487237] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10150.032954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10150.096531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10151.304115] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [10151.329138] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [10151.381528] Lustre: lustre-MDT0000: new disk, initializing [10151.436182] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10151.449150] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10154.298324] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [10161.288540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10161.334337] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10161.402812] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [10161.433973] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [10161.437142] Lustre: Skipped 1 previous similar message [10161.478589] Lustre: lustre-MDT0001: new disk, initializing [10161.539736] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10161.576106] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10161.584851] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10164.514660] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [10171.178140] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10174.527887] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10178.758565] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10178.820812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10178.825523] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [10178.973920] Lustre: lustre-OST0000: new disk, initializing [10178.977560] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [10179.024853] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10181.666819] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [10187.143786] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10187.270265] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10187.274878] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10189.446697] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [10189.585846] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10191.767521] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [10191.878254] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10197.475635] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10197.476088] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10197.477239] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10197.482945] LustreError: Skipped 1 previous similar message [10197.489064] Lustre: Skipped 1 previous similar message [10202.595157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10202.598027] Lustre: Skipped 2 previous similar messages [10207.200478] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [10207.308335] Lustre: server umount lustre-OST0000 complete [10212.834256] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10212.834482] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10212.847212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10212.850103] Lustre: Skipped 1 previous similar message [10215.830312] Lustre: server umount lustre-MDT0000 complete [10217.925282] LustreError: 416046:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761320568 with bad export cookie 14569440515070119865 [10217.926888] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10217.930445] LustreError: 416046:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [10218.075570] Lustre: server umount lustre-MDT0001 complete [10222.242744] Lustre: DEBUG MARKER: == conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================================================== 11:42:52 (1761320572) [10226.752374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10231.052500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10231.264130] Lustre: lustre-MDT0000: new disk, initializing [10231.282618] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10231.327212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10233.328473] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [10236.392095] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10237.659399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10237.840730] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10240.008411] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [10243.878740] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10246.912453] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10248.166185] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10248.169931] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10248.174962] Lustre: Skipped 1 previous similar message [10248.178928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10252.257338] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10252.257632] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10254.235123] Lustre: server umount lustre-MDT0000 complete [10256.261930] LustreError: 420310:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761320607 with bad export cookie 14569440515070120719 [10256.264772] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10256.268293] LustreError: 420310:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [10261.087236] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 11:43:31 (1761320611) [10264.267287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10264.502293] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10264.547853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10266.578869] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [10269.666369] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10271.129051] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10273.510758] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [10277.167329] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10279.875790] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10282.867676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10283.031932] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10283.035178] Lustre: Skipped 1 previous similar message [10284.921781] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [10288.795109] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10298.175347] Lustre: Permanently deactivating lustre-OST0000 [10298.184142] Lustre: Setting parameter lustre-OST0000-osc.osc.active in log lustre-client [10308.124652] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10309.140103] LustreError: 426067:0:(mgs_llog.c:1551:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002020 [10309.147160] LustreError: 426067:0:(mgs_llog.c:1605:mgs_replace_nids()) Only MGS is allowed to be started [10309.152856] LustreError: 426067:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -115 [10310.745619] LustreError: 426116:0:(mgs_llog.c:1605:mgs_replace_nids()) Only MGS is allowed to be started [10310.750375] LustreError: 426116:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -115 [10318.056455] Lustre: server umount lustre-OST0000 complete [10318.057854] Lustre: Skipped 1 previous similar message [10319.306176] LustreError: 426365:0:(mgs_llog.c:1605:mgs_replace_nids()) Only MGS is allowed to be started [10319.309098] LustreError: 426365:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -115 [10322.402433] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10322.410243] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10322.418694] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10322.421287] Lustre: Skipped 1 previous similar message [10326.496170] Lustre: 426462:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761320671/real 1761320671] req@0000000027ced5c7 x1846878379482432/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761320677 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [10326.588961] Lustre: server umount lustre-MDT0000 complete [10328.034239] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10328.040190] LustreError: Skipped 1 previous similar message [10328.507206] LustreError: 422906:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761320679 with bad export cookie 14569440515070121258 [10328.510822] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10328.512906] LustreError: 422906:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [10334.688186] Lustre: 426663:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761320679/real 1761320679] req@0000000027ced5c7 x1846878379483328/t0(0) o9->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761320685 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [10339.174689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10341.192591] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [10342.548212] LustreError: 427572:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -6 [10345.903427] Lustre: 427918:0:(mgs_llog.c:1311:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.156@tcp [10353.121072] Lustre: 428017:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761320698/real 1761320698] req@000000003fd8170d x1846878379483904/t0(0) o251->MGC192.168.202.156@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761320704 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [10353.139708] Lustre: server umount MGS complete [10353.147113] Lustre: Skipped 1 previous similar message [10359.282193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10359.522503] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10359.536512] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10359.539918] Lustre: Skipped 1 previous similar message [10359.580260] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10361.848313] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [10366.453295] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10366.632194] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10368.679213] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [10372.157944] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10374.770701] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10375.726422] Lustre: Permanently reactivating lustre-OST0000 [10375.732866] Lustre: Modifying parameter lustre-OST0000-osc.osc.active in log lustre-client [10375.736233] Lustre: Skipped 2 previous similar messages [10398.689223] LustreError: 430441:0:(osp_dev.c:684:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [10402.919642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10405.221849] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [10408.617414] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10413.539641] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10413.539718] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10413.541256] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10413.541261] Lustre: Skipped 1 previous similar message [10413.544935] Lustre: Skipped 2 previous similar messages [10416.700988] Lustre: server umount lustre-OST0000 complete [10423.265088] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10426.157329] LustreError: 430331:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761320777 with bad export cookie 14569440515070122203 [10426.161177] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10426.162531] LustreError: 430331:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [10431.705434] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [10432.914671] Key type lgssc unregistered [10433.072516] LNet: 433277:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10433.077725] LNet: Removed LNI 192.168.202.156@tcp [10433.459846] Key type .llcrypt unregistered [10433.462030] Key type ._llcrypt unregistered [10442.060261] alg: No test for adler32 (adler32-zlib) [10442.812456] Key type ._llcrypt registered [10442.813620] Key type .llcrypt registered [10442.864601] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [10450.389773] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [10450.934373] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10451.111831] Lustre: Lustre: Build Version: 2.15.7_9_g1c128dc [10451.217139] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [10451.219916] LNet: Accept secure, port 988 [10452.824181] Key type lgssc registered [10453.300118] Lustre: Echo OBD driver; http://www.lustre.org/ [10457.799856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10460.982865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10463.898339] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10467.137548] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10470.543090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10470.576948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10471.710045] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [10471.730256] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [10471.785390] Lustre: lustre-MDT0000: new disk, initializing [10471.844107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10471.854858] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10474.019777] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [10479.445851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10479.479527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10479.526266] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [10479.548843] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [10479.551881] Lustre: Skipped 1 previous similar message [10479.592479] Lustre: lustre-MDT0001: new disk, initializing [10479.633580] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10479.653275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10479.657688] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10481.519389] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [10486.787603] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10489.631855] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10492.749697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10492.784401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10492.788877] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [10492.915360] Lustre: lustre-OST0000: new disk, initializing [10492.918505] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [10492.967067] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10494.827941] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [10499.770361] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10501.654706] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [10502.659389] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10502.664130] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10502.791687] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [10504.749702] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [10504.840137] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10507.748820] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10507.749089] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10507.758979] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10507.763716] LustreError: Skipped 1 previous similar message [10507.765714] Lustre: Skipped 1 previous similar message [10511.854350] Lustre: server umount lustre-OST0000 complete [10515.426331] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10515.430332] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10515.435581] Lustre: Skipped 1 previous similar message [10515.438371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10517.985051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10517.985626] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10523.106247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10523.109092] Lustre: Skipped 2 previous similar messages [10528.224615] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [10528.225703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10528.231135] Lustre: Skipped 1 previous similar message [10528.278130] Lustre: server umount lustre-MDT0000 complete [10530.072488] LustreError: 436544:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761320881 with bad export cookie 11953410495718175889 [10530.074870] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10530.078987] LustreError: 436544:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [10530.249076] Lustre: server umount lustre-MDT0001 complete [10534.457381] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [10535.391211] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 11:48:05 (1761320885) [10538.740443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10538.961405] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10539.011669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10540.975978] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [10544.097629] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10544.695790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10544.862582] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10546.703861] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [10550.291052] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10552.927751] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10556.442402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10556.588227] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10558.531354] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [10561.872492] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10562.729434] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [10573.543557] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [10576.353547] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10576.357284] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10576.363665] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10580.452419] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10581.858332] Lustre: server umount lustre-OST0000 complete [10589.714987] Lustre: server umount lustre-MDT0000 complete [10590.689205] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10590.694901] LustreError: Skipped 1 previous similar message [10591.258358] LustreError: 441137:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761320942 with bad export cookie 11953410495718176722 [10591.260383] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10591.263737] LustreError: 441137:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [10596.871726] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [10598.058228] Key type lgssc unregistered [10598.192886] LNet: 445019:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10598.197481] LNet: Removed LNI 192.168.202.156@tcp [10598.556489] Key type .llcrypt unregistered [10598.558192] Key type ._llcrypt unregistered [10609.586926] alg: No test for adler32 (adler32-zlib) [10610.342029] Key type ._llcrypt registered [10610.344232] Key type .llcrypt registered [10610.424125] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [10610.563667] Key type .llcrypt unregistered [10610.565792] Key type ._llcrypt unregistered [10619.013964] alg: No test for adler32 (adler32-zlib) [10619.765510] Key type ._llcrypt registered [10619.767093] Key type .llcrypt registered [10619.826524] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [10620.241854] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10620.504197] Lustre: Lustre: Build Version: 2.15.7_9_g1c128dc [10620.602704] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [10620.606849] LNet: Accept secure, port 988 [10622.208237] Key type lgssc registered [10622.716111] Lustre: Echo OBD driver; http://www.lustre.org/ [10626.427961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10627.636704] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10627.660364] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10629.577701] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [10632.673595] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10633.171270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10633.324494] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10635.046730] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [10638.138668] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10638.817621] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10638.823426] LustreError: Skipped 3 previous similar messages [10640.903436] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10641.078554] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10642.095926] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [10642.913846] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [10646.497384] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10646.501633] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10646.507774] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10647.522709] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10647.523187] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10647.531298] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10652.454513] Lustre: server umount lustre-OST0000 complete [10656.738914] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10656.741396] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10656.742605] LustreError: Skipped 1 previous similar message [10656.752185] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10656.754811] Lustre: Skipped 1 previous similar message [10660.311569] Lustre: server umount lustre-MDT0000 complete [10661.980358] LustreError: 449098:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761321012 with bad export cookie 15956370955539878595 [10661.982595] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10661.985051] LustreError: 449098:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [10662.112407] Lustre: server umount lustre-MDT0001 complete [10667.837096] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [10669.201863] Key type lgssc unregistered [10669.347626] LNet: 450603:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10669.352434] LNet: Removed LNI 192.168.202.156@tcp [10669.726743] Key type .llcrypt unregistered [10669.732183] Key type ._llcrypt unregistered [10681.249367] alg: No test for adler32 (adler32-zlib) [10682.001370] Key type ._llcrypt registered [10682.002575] Key type .llcrypt registered [10682.053777] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [10682.525163] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10682.701205] Lustre: Lustre: Build Version: 2.15.7_9_g1c128dc [10682.799233] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [10682.801806] LNet: Accept secure, port 988 [10684.392163] Key type lgssc registered [10684.843717] Lustre: Echo OBD driver; http://www.lustre.org/ [10688.590915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10705.120334] LustreError: 452039:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.156@tcp: failed processing log, type 1: rc = -5 [10736.864207] LustreError: 452039:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.156@tcp: failed processing log, type 4: rc = -110 [10767.584496] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10769.633854] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [10773.279535] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10776.213989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10789.215708] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10789.247984] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10789.276452] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:97 [10791.100121] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [10794.466084] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10794.494665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10794.644519] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10796.427864] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [10799.748905] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10802.342274] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10805.219187] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10805.219427] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10805.223696] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10805.230103] Lustre: Skipped 1 previous similar message [10810.339101] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10810.341870] Lustre: Skipped 1 previous similar message [10811.246777] Lustre: server umount lustre-OST0000 complete [10815.457350] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10815.457802] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10815.471082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10815.473912] Lustre: Skipped 2 previous similar messages [10819.159694] Lustre: server umount lustre-MDT0000 complete [10820.576962] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10820.582502] LustreError: Skipped 1 previous similar message [10820.835834] LustreError: 452045:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761321171 with bad export cookie 868090365467393116 [10820.836585] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10820.840710] LustreError: 452045:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [10820.971615] Lustre: server umount lustre-MDT0001 complete [10826.398845] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [10827.570486] Key type lgssc unregistered [10827.707596] LNet: 455730:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10827.711558] LNet: Removed LNI 192.168.202.156@tcp [10828.065258] Key type .llcrypt unregistered [10828.067060] Key type ._llcrypt unregistered [10839.551358] alg: No test for adler32 (adler32-zlib) [10840.303509] Key type ._llcrypt registered [10840.305033] Key type .llcrypt registered [10840.353064] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [10840.824124] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10840.997956] Lustre: Lustre: Build Version: 2.15.7_9_g1c128dc [10841.094779] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [10841.096905] LNet: Accept secure, port 988 [10842.688164] Key type lgssc registered [10843.210207] Lustre: Echo OBD driver; http://www.lustre.org/ [10846.728696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10847.912836] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10847.937545] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10849.673921] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [10853.141545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10853.317259] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10855.004773] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [10858.083302] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10860.451680] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10863.074670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10863.207822] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10864.230599] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:129 [10864.866965] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [10867.992789] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10873.826815] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10873.827740] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10873.829121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10873.833496] Lustre: Skipped 1 previous similar message [10874.485254] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.56@tcp (stopping) [10874.488391] Lustre: Skipped 2 previous similar messages [10876.254382] Lustre: server umount lustre-MDT0000 complete [10877.254610] LustreError: 457171:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.56@tcp arrived at 1761321228 with bad export cookie 8288614428181523579 [10877.262055] LustreError: 457171:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [10878.072758] LustreError: 457171:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761321229 with bad export cookie 8288614428181522732 [10878.076823] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10878.078925] LustreError: 457171:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [10878.945063] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10878.945636] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10878.945728] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10878.951164] LustreError: Skipped 1 previous similar message [10878.957664] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10884.067237] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10884.067369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10884.072463] LustreError: Skipped 1 previous similar message [10884.192134] Lustre: server umount lustre-OST0000 complete [10887.227721] Lustre: server umount lustre-MDT0001 complete [10892.730446] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [10893.959607] Key type lgssc unregistered [10894.091624] LNet: 460992:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10894.096628] LNet: Removed LNI 192.168.202.156@tcp [10894.450281] Key type .llcrypt unregistered [10894.452752] Key type ._llcrypt unregistered [10904.904076] alg: No test for adler32 (adler32-zlib) [10905.656410] Key type ._llcrypt registered [10905.657988] Key type .llcrypt registered [10905.706784] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [10906.228648] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10906.389624] Lustre: Lustre: Build Version: 2.15.7_9_g1c128dc [10906.477656] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [10906.479937] LNet: Accept secure, port 988 [10908.064244] Key type lgssc registered [10908.504781] Lustre: Echo OBD driver; http://www.lustre.org/ [10911.638245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10912.809630] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10912.832531] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10914.612579] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [10917.857887] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10918.409480] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10918.583524] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10920.415659] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [10923.626348] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10925.889404] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10928.538750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10928.674085] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10930.371226] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [10933.376820] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10943.019698] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:161 [10948.068052] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10948.075845] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10948.078871] Lustre: Skipped 1 previous similar message [10949.238162] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.56@tcp (stopping) [10951.429824] Lustre: server umount lustre-MDT0001 complete [10953.186908] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10953.193304] LustreError: Skipped 1 previous similar message [10958.305116] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10958.305687] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10958.305759] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10958.319866] Lustre: Skipped 1 previous similar message [10958.322701] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10959.659296] Lustre: server umount lustre-OST0000 complete [10967.520184] Lustre: 465527:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761321312/real 1761321312] req@00000000837044d3 x1846879198393280/t0(0) o251->MGC192.168.202.156@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761321318 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [10967.590096] Lustre: server umount lustre-MDT0000 complete [10973.956715] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [10975.155990] Key type lgssc unregistered [10975.283566] LNet: 466282:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10975.287310] LNet: Removed LNI 192.168.202.156@tcp [10975.639882] Key type .llcrypt unregistered [10975.641521] Key type ._llcrypt unregistered [10986.476783] alg: No test for adler32 (adler32-zlib) [10987.228380] Key type ._llcrypt registered [10987.229883] Key type .llcrypt registered [10987.280419] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [10993.388561] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [10993.749957] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10994.010336] Lustre: Lustre: Build Version: 2.15.7_9_g1c128dc [10994.108964] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [10994.111383] LNet: Accept secure, port 988 [10995.696169] Key type lgssc registered [10996.166480] Lustre: Echo OBD driver; http://www.lustre.org/ [11000.178710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11003.262792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11005.847799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11008.583208] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11012.232078] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [11016.036857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11018.736628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11018.766692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11019.873392] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [11019.888245] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [11019.929810] Lustre: lustre-MDT0000: new disk, initializing [11019.965077] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11019.974774] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11021.797471] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [11025.959883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11028.849206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11028.887559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11028.891487] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [11029.001721] Lustre: lustre-OST0000: new disk, initializing [11029.004942] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [11029.008039] Lustre: Skipped 1 previous similar message [11029.040026] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11030.842964] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [11035.346336] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11039.011757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11041.701573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11041.732590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11041.774724] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [11041.796330] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [11041.837377] Lustre: lustre-MDT0001: new disk, initializing [11041.863385] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [11041.867348] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [11041.885349] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11043.678890] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [11047.697540] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 40 [11047.780645] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [11049.372482] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 40 [11049.459203] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [11053.538347] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11053.538432] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11053.544405] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11053.549703] Lustre: Skipped 1 previous similar message [11057.410880] Lustre: server umount lustre-MDT0001 complete [11058.658039] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11058.663639] LustreError: Skipped 1 previous similar message [11063.778429] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11063.783895] LustreError: Skipped 2 previous similar messages [11068.898928] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11068.904428] LustreError: Skipped 1 previous similar message [11070.432945] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11070.436994] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11070.443013] Lustre: Skipped 1 previous similar message [11070.448481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11070.455547] Lustre: Skipped 1 previous similar message [11074.017678] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11074.017869] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11076.193828] Lustre: server umount lustre-OST0000 complete [11089.376237] Lustre: 474681:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761321434/real 1761321434] req@00000000402535ee x1846879290670656/t0(0) o251->MGC192.168.202.156@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761321440 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [11089.450929] Lustre: server umount lustre-MDT0000 complete [11095.840966] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [11096.915322] Key type lgssc unregistered [11097.028462] LNet: 475435:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11097.031910] LNet: Removed LNI 192.168.202.156@tcp [11097.336951] Key type .llcrypt unregistered [11097.338238] Key type ._llcrypt unregistered [11113.164370] alg: No test for adler32 (adler32-zlib) [11113.917478] Key type ._llcrypt registered [11113.919074] Key type .llcrypt registered [11113.970153] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [11114.089218] Key type .llcrypt unregistered [11114.090720] Key type ._llcrypt unregistered [11121.460515] alg: No test for adler32 (adler32-zlib) [11122.213417] Key type ._llcrypt registered [11122.214613] Key type .llcrypt registered [11122.270116] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [11122.626561] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11122.862182] Lustre: Lustre: Build Version: 2.15.7_9_g1c128dc [11122.947297] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [11122.950142] LNet: Accept secure, port 988 [11124.536120] Key type lgssc registered [11124.918331] Lustre: Echo OBD driver; http://www.lustre.org/ [11127.846617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11130.190878] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11132.577742] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11135.324913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11135.357737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11136.452823] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [11136.464639] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [11136.504540] Lustre: lustre-MDT0000: new disk, initializing [11136.537533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11136.548224] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11138.192846] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [11142.884327] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11142.914974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11142.960691] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [11142.985380] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [11142.988865] Lustre: Skipped 1 previous similar message [11143.032717] Lustre: lustre-MDT0001: new disk, initializing [11143.073738] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11143.090267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11143.094399] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11144.827908] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [11149.275758] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11151.528810] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11154.452086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11154.488557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11154.493096] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [11154.617536] Lustre: lustre-OST0000: new disk, initializing [11154.620248] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [11154.666292] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11156.530301] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [11160.998240] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11161.093476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11161.098533] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11163.617203] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11163.620282] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11163.625321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11166.177328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11166.178265] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11166.180040] Lustre: Skipped 1 previous similar message [11166.185279] Lustre: Skipped 1 previous similar message [11169.737458] Lustre: server umount lustre-MDT0000 complete [11171.220655] LustreError: 478917:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761321522 with bad export cookie 5271934454870112823 [11171.222174] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11171.226364] LustreError: 478917:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [11171.297586] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11171.297961] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11171.316350] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11171.321989] Lustre: Skipped 1 previous similar message [11171.394843] Lustre: server umount lustre-MDT0001 complete [11178.988977] Lustre: server umount lustre-OST0000 complete [11182.970787] Lustre: DEBUG MARKER: == conf-sanity test 73: failnode to update from mountdata properly ========================================================== 11:58:53 (1761321533) [11183.455449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11186.388120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11186.610850] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11186.639400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11188.516639] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [11191.777621] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11192.110382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11192.265052] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11193.983581] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [11196.985806] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11199.235417] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11201.761898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11201.792522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11201.867239] Lustre: Found index 0 for lustre-OST0000, updating log [11201.925529] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11203.568842] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [11206.405209] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11216.294297] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5 to 0x0:33 [11221.477399] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11221.479572] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11221.484767] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11221.492432] Lustre: Skipped 1 previous similar message [11224.812262] Lustre: server umount lustre-OST0000 complete [11226.593187] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11226.593738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11226.597627] Lustre: Skipped 2 previous similar messages [11232.521769] Lustre: server umount lustre-MDT0000 complete [11233.901069] LustreError: 484947:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761321584 with bad export cookie 5271934454870114153 [11233.903532] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11233.905278] LustreError: 484947:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [11237.180166] Lustre: DEBUG MARKER: == conf-sanity test 75: The order of --index should be irrelevant ========================================================== 11:59:47 (1761321587) [11239.268939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11241.146813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11243.208225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11245.158567] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11251.580386] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [11255.008788] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [11258.914208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11261.545052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11264.057073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11266.635262] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11269.369858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11269.391791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11269.485845] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [11269.499260] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [11269.540561] Lustre: lustre-MDT0000: new disk, initializing [11269.572930] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11269.581228] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11271.268062] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [11276.351742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11276.386310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11276.435305] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [11276.462274] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [11276.465073] Lustre: Skipped 1 previous similar message [11276.515446] Lustre: lustre-MDT0001: new disk, initializing [11276.580884] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11276.585651] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11278.677757] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [11283.523026] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11286.228688] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11289.254939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11289.294751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11289.299290] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [11289.424812] Lustre: lustre-OST0000: new disk, initializing [11289.429451] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [11291.271317] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all 8 [11291.782917] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11291.788301] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11295.779883] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11297.548829] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [11297.642253] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11299.385244] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [11299.478304] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11302.370973] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11302.371437] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11302.373611] LustreError: Skipped 1 previous similar message [11302.378834] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11302.381956] Lustre: Skipped 4 previous similar messages [11306.339552] Lustre: server umount lustre-OST0000 complete [11306.341352] Lustre: Skipped 1 previous similar message [11312.610064] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11318.753349] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11318.755515] Lustre: Skipped 5 previous similar messages [11322.337017] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [11322.388184] LustreError: 493709:0:(ldlm_resource.c:1127:ldlm_resource_complain()) MGS: namespace resource [0x736d61726170:0x3:0x0].0x0 (00000000cb1de141) refcount nonzero (2) after lock cleanup; forcing cleanup. [11322.402423] Lustre: server umount lustre-MDT0000 complete [11323.874354] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11323.880091] LustreError: Skipped 1 previous similar message [11324.087539] LustreError: 490028:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761321675 with bad export cookie 5271934454870115217 [11324.088357] LustreError: 166-1: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11324.092948] LustreError: 490028:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [11333.834129] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [11337.707071] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [11342.088352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11345.120118] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11347.502056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11350.001727] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11350.617838] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 11159 sec ========= 12:01:41 (1761321701) [11365.710228] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [11366.733419] Key type lgssc unregistered [11366.841359] LNet: 498765:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11366.844080] LNet: Removed LNI 192.168.202.156@tcp [11367.144069] Key type .llcrypt unregistered [11367.145311] Key type ._llcrypt unregistered