[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 3.0.0 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-2.fc40 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 532575953 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f53f0-0x000f53ff] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5200 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1D87 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1C23 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001BE3 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1C97 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1D27 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1D5F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1c23-0xbffe1c96] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1c22] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1c97-0xbffe1d26] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1d27-0xbffe1d5e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1d5f-0xbffe1d86] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001016] APIC: Switch to symmetric I/O mode setup [ 0.002000] x2apic enabled [ 0.002016] Switched APIC routing to physical x2apic. [ 0.003019] kvm-guest: setup PV IPIs [ 0.006000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.006000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.006031] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.007904] pid_max: default: 32768 minimum: 301 [ 0.009057] LSM: Security Framework initializing [ 0.010157] Yama: becoming mindful. [ 0.011059] SELinux: Initializing. [ 0.012078] *** VALIDATE selinux *** [ 0.019511] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024577] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025259] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026133] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027141] *** VALIDATE tmpfs *** [ 0.029165] *** VALIDATE proc *** [ 0.030000] *** VALIDATE cgroup *** [ 0.030022] *** VALIDATE cgroup2 *** [ 0.032283] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034058] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036042] Spectre V2 : User space: Vulnerable [ 0.037018] Speculative Store Bypass: Vulnerable [ 0.040904] debug: unmapping init [mem 0xffffffff85e59000-0xffffffff85e60fff] [ 0.045140] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046759] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047048] ... version: 2 [ 0.048032] ... bit width: 48 [ 0.049021] ... generic registers: 4 [ 0.050018] ... value mask: 0000ffffffffffff [ 0.051023] ... max period: 00007fffffffffff [ 0.052020] ... fixed-purpose events: 3 [ 0.053013] ... event mask: 000000070000000f [ 0.054531] rcu: Hierarchical SRCU implementation. [ 0.056851] smp: Bringing up secondary CPUs ... [ 0.059068] x86: Booting SMP configuration: [ 0.060503] .... node #0, CPUs: #1 #2 #3 [ 0.078074] smp: Brought up 1 node, 4 CPUs [ 0.080014] smpboot: Max logical packages: 1 [ 0.081012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.146643] node 0 deferred pages initialised in 62ms [ 0.166335] devtmpfs: initialized [ 0.167291] x86/mm: Memory block size: 128MB [ 0.170154] gcov: version magic: 0x41383552 [ 0.172255] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.173071] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.174275] pinctrl core: initialized pinctrl subsystem [ 0.175452] [ 0.176017] ************************************************************* [ 0.177020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.178019] ** ** [ 0.179019] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.180018] ** ** [ 0.181017] ** This means that this kernel is built to expose internal ** [ 0.182011] ** IOMMU data structures, which may compromise security on ** [ 0.183021] ** your system. ** [ 0.184018] ** ** [ 0.185015] ** If you see this message and you are not debugging the ** [ 0.186021] ** kernel, report this immediately to your vendor! ** [ 0.187023] ** ** [ 0.188018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.189018] ************************************************************* [ 0.191000] NET: Registered protocol family 16 [ 0.193894] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.198100] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.202096] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.209119] cpuidle: using governor menu [ 0.211000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.223076] PCI: Using configuration type 1 for base access [ 0.225157] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.243323] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.246339] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.253049] cryptd: max_cpu_qlen set to 1000 [ 0.257388] ACPI: Added _OSI(Module Device) [ 0.259017] ACPI: Added _OSI(Processor Device) [ 0.261018] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.263020] ACPI: Added _OSI(Processor Aggregator Device) [ 0.267700] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.275491] ACPI: Interpreter enabled [ 0.277058] ACPI: PM: (supports S0 S3 S4 S5) [ 0.279013] ACPI: Using IOAPIC for interrupt routing [ 0.280000] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.282421] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.292202] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.294061] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.297027] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.300101] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.305507] acpiphp: Slot [2] registered [ 0.307185] acpiphp: Slot [5] registered [ 0.308151] acpiphp: Slot [6] registered [ 0.310118] acpiphp: Slot [7] registered [ 0.311174] acpiphp: Slot [8] registered [ 0.312113] acpiphp: Slot [9] registered [ 0.314128] acpiphp: Slot [10] registered [ 0.315138] acpiphp: Slot [3] registered [ 0.316106] acpiphp: Slot [4] registered [ 0.318095] acpiphp: Slot [11] registered [ 0.319098] acpiphp: Slot [12] registered [ 0.320108] acpiphp: Slot [13] registered [ 0.322111] acpiphp: Slot [14] registered [ 0.323141] acpiphp: Slot [15] registered [ 0.324103] acpiphp: Slot [16] registered [ 0.326101] acpiphp: Slot [17] registered [ 0.327084] acpiphp: Slot [18] registered [ 0.329106] acpiphp: Slot [19] registered [ 0.330117] acpiphp: Slot [20] registered [ 0.331094] acpiphp: Slot [21] registered [ 0.333124] acpiphp: Slot [22] registered [ 0.334103] acpiphp: Slot [23] registered [ 0.335086] acpiphp: Slot [24] registered [ 0.337097] acpiphp: Slot [25] registered [ 0.338103] acpiphp: Slot [26] registered [ 0.339096] acpiphp: Slot [27] registered [ 0.341135] acpiphp: Slot [28] registered [ 0.342096] acpiphp: Slot [29] registered [ 0.344094] acpiphp: Slot [30] registered [ 0.345107] acpiphp: Slot [31] registered [ 0.346052] PCI host bridge to bus 0000:00 [ 0.347022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.350024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.352023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.355027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.357025] pci_bus 0000:00: root bus resource [mem 0x380000000000-0x38007fffffff window] [ 0.361034] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.363190] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.368289] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.370000] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.379014] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.384486] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.387022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.389019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.391027] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.394654] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.397925] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.400062] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.403993] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.407960] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.420018] pci 0000:00:02.0: reg 0x20: [mem 0x380000000000-0x380000003fff 64bit pref] [ 0.425000] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.430783] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.439016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.444018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.463021] pci 0000:00:05.0: reg 0x20: [mem 0x380000004000-0x380000007fff 64bit pref] [ 0.475826] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.489022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.501020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.517019] pci 0000:00:06.0: reg 0x20: [mem 0x380000008000-0x38000000bfff 64bit pref] [ 0.530579] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.537016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.542015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.554018] pci 0000:00:07.0: reg 0x20: [mem 0x38000000c000-0x38000000ffff 64bit pref] [ 0.561953] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.568016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.574017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.585021] pci 0000:00:08.0: reg 0x20: [mem 0x380000010000-0x380000013fff 64bit pref] [ 0.599014] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.610023] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.616016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.636035] pci 0000:00:09.0: reg 0x20: [mem 0x380000014000-0x380000017fff 64bit pref] [ 0.649654] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.655000] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.655027] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.667039] pci 0000:00:0a.0: reg 0x20: [mem 0x380000018000-0x38000001bfff 64bit pref] [ 0.677457] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.681388] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.686381] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.689123] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.694255] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.698149] iommu: Default domain type: Passthrough [ 0.701364] SCSI subsystem initialized [ 0.702143] ACPI: bus type USB registered [ 0.705107] usbcore: registered new interface driver usbfs [ 0.708151] usbcore: registered new interface driver hub [ 0.711135] usbcore: registered new device driver usb [ 0.713190] pps_core: LinuxPPS API ver. 1 registered [ 0.714000] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.714000] PTP clock support registered [ 0.714376] EDAC MC: Ver: 3.0.0 [ 0.716108] PCI: Using ACPI for IRQ routing [ 0.720124] NetLabel: Initializing [ 0.721010] NetLabel: domain hash size = 128 [ 0.723017] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.726097] NetLabel: unlabeled traffic allowed by default [ 0.727468] vgaarb: loaded [ 0.730315] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.733013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.741600] clocksource: Switched to clocksource kvm-clock [ 0.922097] VFS: Disk quotas dquot_6.6.0 [ 0.924118] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.929145] *** VALIDATE ramfs *** [ 0.932246] *** VALIDATE hugetlbfs *** [ 0.934276] pnp: PnP ACPI init [ 0.938307] pnp: PnP ACPI: found 6 devices [ 0.965997] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.971146] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.976732] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.979422] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.981935] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.985854] pci_bus 0000:00: resource 8 [mem 0x380000000000-0x38007fffffff window] [ 0.991325] NET: Registered protocol family 2 [ 0.999839] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.010813] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.017725] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.028634] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.032704] TCP: Hash tables configured (established 65536 bind 65536) [ 1.036654] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.043279] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.046405] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.049609] NET: Registered protocol family 1 [ 1.053563] RPC: Registered named UNIX socket transport module. [ 1.056096] RPC: Registered udp transport module. [ 1.058229] RPC: Registered tcp transport module. [ 1.060140] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.063177] NET: Registered protocol family 44 [ 1.064931] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.067041] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.071906] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.076175] PCI: CLS 0 bytes, default 64 [ 1.079249] Unpacking initramfs... [ 2.751797] debug: unmapping init [mem 0xffff90e2bcc54000-0xffff90e2bffbffff] [ 2.759719] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.762257] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.766644] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.296183] Initialise system trusted keyrings [ 3.297937] Key type blacklist registered [ 3.300551] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.310143] zbud: loaded [ 3.313610] *** VALIDATE nfs *** [ 3.315035] *** VALIDATE nfs4 *** [ 3.316711] pstore: using deflate compression [ 3.320536] Platform Keyring initialized [ 3.462297] NET: Registered protocol family 38 [ 3.463727] Key type asymmetric registered [ 3.465263] Asymmetric key parser 'x509' registered [ 3.467192] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.470729] io scheduler mq-deadline registered [ 3.472688] io scheduler kyber registered [ 3.474712] io scheduler bfq registered [ 3.476798] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.481783] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.490498] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.493198] ACPI: Power Button [PWRF] [ 3.599224] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.703619] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.920339] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.013848] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.355114] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.396497] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.432427] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.453871] Non-volatile memory driver v1.3 [ 4.455721] Linux agpgart interface v0.103 [ 4.516607] virtio_blk virtio1: [vda] 67992 512-byte logical blocks (34.8 MB/33.2 MiB) [ 4.523870] vda: detected capacity change from 0 to 34811904 [ 4.557497] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.562794] vdb: detected capacity change from 0 to 1073741824 [ 4.586307] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.590132] vdc: detected capacity change from 0 to 2621440000 [ 4.615399] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.620178] vdd: detected capacity change from 0 to 2621440000 [ 4.646926] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.651525] vde: detected capacity change from 0 to 4294967296 [ 4.690346] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.695354] vdf: detected capacity change from 0 to 4294967296 [ 4.733563] libphy: Fixed MDIO Bus: probed [ 4.750180] usbcore: registered new interface driver usbserial_generic [ 4.764845] usbserial: USB Serial support registered for generic [ 4.771410] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.778564] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.781274] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.785865] mousedev: PS/2 mouse device common for all mice [ 4.791596] rtc_cmos 00:05: RTC can wake from S4 [ 4.796339] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.797294] rtc_cmos 00:05: registered as rtc0 [ 4.802787] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.810268] intel_pstate: CPU model not supported [ 4.812769] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.823933] hid: raw HID events driver (C) Jiri Kosina [ 4.830480] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.834222] usbcore: registered new interface driver usbhid [ 4.834231] usbhid: USB HID core driver [ 4.834366] drop_monitor: Initializing network drop monitor service [ 4.834511] Initializing XFRM netlink socket [ 4.834920] NET: Registered protocol family 10 [ 4.840706] Segment Routing with IPv6 [ 4.840767] NET: Registered protocol family 17 [ 4.842429] mpls_gso: MPLS GSO support [ 4.866717] RAS: Correctable Errors collector initialized. [ 4.868800] AVX version of gcm_enc/dec engaged. [ 4.870931] AES CTR mode by8 optimization enabled [ 5.131859] sched_clock: Marking stable (5131359179, 0)->(6672440359, -1541081180) [ 5.136258] registered taskstats version 1 [ 5.140813] Loading compiled-in X.509 certificates [ 5.143292] zswap: loaded using pool lzo/zbud [ 5.182616] Key type big_key registered [ 5.202381] Key type encrypted registered [ 5.203822] ima: No TPM chip found, activating TPM-bypass! [ 5.209655] ima: Allocated hash algorithm: sha1 [ 5.211237] ima: No architecture policies found [ 5.215857] evm: Initialising EVM extended attributes: [ 5.217682] evm: security.selinux [ 5.219228] evm: security.ima [ 5.220551] evm: security.capability [ 5.221913] evm: HMAC attrs: 0x1 [ 5.227455] rtc_cmos 00:05: setting system clock to 2025-10-24 16:02:51 UTC (1761321771) [ 5.237618] debug: unmapping init [mem 0xffffffff86e03000-0xffffffff86ffffff] [ 5.241383] debug: unmapping init [mem 0xffffffff85b82000-0xffffffff85e58fff] [ 5.254065] Write protecting the kernel read-only data: 28672k [ 5.262075] debug: unmapping init [mem 0xffffffff84203000-0xffffffff843fffff] [ 5.265931] debug: unmapping init [mem 0xffffffff84b14000-0xffffffff84bfffff] [ 5.306293] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.330712] systemd[1]: Detected virtualization kvm. [ 5.334179] systemd[1]: Detected architecture x86-64. [ 5.336406] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.378693] systemd[1]: No hostname configured. [ 5.380384] systemd[1]: Set hostname to . [ 5.382738] random: systemd: uninitialized urandom read (16 bytes read) [ 5.385216] systemd[1]: Initializing machine ID from random generator. [ 5.502341] random: ln: uninitialized urandom read (6 bytes read) [ 5.663893] random: systemd: uninitialized urandom read (16 bytes read) [ 5.667416] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 5.674831] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 5.682667] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... Starting Journal Service... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.077765] device-mapper: uevent: version 1.0.3 [ 7.080967] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 8.428458] random: fast init done [ 8.517529] virtio_net virtio0 ens2: renamed from eth0 [ 8.701177] scsi host0: ata_piix [ 8.715458] scsi host1: ata_piix [ 8.716875] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 8.723754] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 13.730409] random: crng init done [ 13.731972] random: 7 urandom warning(s) missed due to ratelimiting [ 14.083867] dracut-initqueue[586]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 15.686120] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 17.512485] printk: systemd: 26 output lines suppressed due to ratelimiting [ 18.041374] SELinux: Disabled at runtime. [ 18.134656] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 18.142559] systemd[1]: Detected virtualization kvm. [ 18.144351] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 19.128588] systemd[1]: initrd-switch-root.service: Succeeded. [ 19.132290] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 19.142041] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 19.159200] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 19.168292] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 19.184306] systemd[1]: Starting Journal Service... Starting Journal Service... [ 19.210463] systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target rpc_pipefs.target. Mounting Kernel Debug File System... [ OK ] Reached target Slices. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Kernel Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Mounting POSIX Message Queue File System... [ OK ] Reached target Local Encrypted Volumes. [ 19.611372] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Huge Pages File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Initrd Root File System. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 20.503588] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 21.173127] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 21.224474] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 21.948769] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 21.995447] EDAC sbridge: Ver: 1.1.2 [ 25.160985] Key type dns_resolver registered [ 25.621223] NFS: Registering the id_resolver key type [ 25.627236] Key type id_resolver registered [ 25.632137] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (6s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started dnf makecache --timer. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg127-server login: [ 52.383902] libcfs: loading out-of-tree module taints kernel. [ 52.403123] alg: No test for adler32 (adler32-zlib) [ 53.155408] Key type ._llcrypt registered [ 53.157288] Key type .llcrypt registered [ 53.208490] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [ 61.561595] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 62.191474] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 62.486484] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 62.804716] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 62.806857] LNet: Accept secure, port 988 [ 64.432137] Key type lgssc registered [ 65.058227] Lustre: Echo OBD driver; http://www.lustre.org/ [ 72.958757] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 74.481887] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 79.340036] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 82.932190] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 86.696116] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 93.536349] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 99.108962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 99.145451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 100.281747] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 100.296984] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 100.353151] Lustre: lustre-MDT0000: new disk, initializing [ 100.414436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 100.427797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 102.214322] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 108.384253] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 108.420231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 108.459240] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 108.475079] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 108.477759] Lustre: Skipped 1 previous similar message [ 108.523187] Lustre: lustre-MDT0001: new disk, initializing [ 108.550701] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 108.565494] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 108.569914] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 110.285968] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 116.059113] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 116.087757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 116.091290] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 116.202272] Lustre: lustre-OST0000: new disk, initializing [ 116.206159] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 116.235982] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 117.824771] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 123.775943] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 123.809599] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 123.813765] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 123.858645] Lustre: lustre-OST0001: new disk, initializing [ 123.862230] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 123.875924] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 123.880152] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 123.888899] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 125.559377] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 128.999716] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 129.004047] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 132.044048] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 136.392846] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 142.998853] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing check_logdir /tmp/testlogs/ [ 144.938935] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing yml_node [ 147.020399] Lustre: DEBUG MARKER: Client: 2.15.7.12 [ 148.231829] Lustre: DEBUG MARKER: MDS: 2.15.7.12 [ 149.309856] Lustre: DEBUG MARKER: OSS: 2.15.7.12 [ 150.016031] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Oct 24 12:05:14 EDT 2025 [ 153.668403] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball 110 [ 154.282722] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 [ 169.953389] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 169.953594] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 169.953816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 169.959395] Lustre: Skipped 1 previous similar message [ 172.371431] Lustre: server umount lustre-MDT0000 complete [ 173.997872] LustreError: 6260:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761321940 with bad export cookie 13026536000226556500 [ 173.999438] LustreError: 166-1: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 174.004960] LustreError: 6260:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 175.072890] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 175.072977] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 175.074127] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 175.074133] Lustre: Skipped 3 previous similar messages [ 175.080086] LustreError: Skipped 1 previous similar message [ 175.091556] Lustre: Skipped 3 previous similar messages [ 178.080630] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 178.080791] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 178.083237] Lustre: Skipped 2 previous similar messages [ 178.092206] LustreError: Skipped 1 previous similar message [ 180.129327] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 180.129394] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 180.137890] LustreError: Skipped 1 previous similar message [ 180.140278] Lustre: Skipped 1 previous similar message [ 185.312870] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 185.313160] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 185.316164] Lustre: Skipped 1 previous similar message [ 185.321395] LustreError: Skipped 1 previous similar message [ 188.384159] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 188.520944] Lustre: server umount lustre-MDT0001 complete [ 196.409203] Lustre: server umount lustre-OST0000 complete [ 204.479152] Lustre: server umount lustre-OST0001 complete [ 207.445122] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [ 210.894818] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 214.970754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 217.748609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 220.310911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 222.904752] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 226.053336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 226.109409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 226.214499] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 226.226318] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 226.265604] Lustre: lustre-MDT0000: new disk, initializing [ 226.305759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 226.320961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 228.092674] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 233.429065] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 233.459836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 233.505263] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 233.624889] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 235.678610] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 240.737040] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 243.293972] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 246.086460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 246.114366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 246.118820] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 246.224544] Lustre: lustre-OST0000: new disk, initializing [ 246.226350] Lustre: Skipped 1 previous similar message [ 246.228414] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 246.231518] Lustre: Skipped 2 previous similar messages [ 246.263278] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 246.266488] Lustre: Skipped 1 previous similar message [ 247.553545] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 247.557378] Lustre: Skipped 1 previous similar message [ 247.559995] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 248.021798] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 252.815473] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 254.756152] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 254.855766] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 256.715848] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 256.808722] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 258.018472] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 258.018714] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 258.022018] LustreError: Skipped 1 previous similar message [ 258.028485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 258.034017] Lustre: Skipped 1 previous similar message [ 263.757193] Lustre: server umount lustre-OST0000 complete [ 268.258669] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 268.263393] Lustre: Skipped 2 previous similar messages [ 273.295680] LustreError: 14301:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761322039 with bad export cookie 13026536000226557872 [ 273.298807] LustreError: 166-1: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 273.300620] LustreError: 14301:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 273.459653] Lustre: server umount lustre-MDT0001 complete [ 273.461500] Lustre: Skipped 1 previous similar message [ 275.700996] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 12:07:20 (1761322040) [ 278.786741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 279.039804] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 279.085373] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 281.233519] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 285.270886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 287.540882] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 290.887077] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 293.203873] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 296.054973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 298.042353] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 301.430279] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 314.163939] Lustre: Modifying parameter general.osc.*.max_dirty_mb in log params [ 319.971552] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 319.972865] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 319.976421] Lustre: Skipped 2 previous similar messages [ 319.978333] Lustre: Skipped 5 previous similar messages [ 323.793196] Lustre: server umount lustre-MDT0000 complete [ 325.089564] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 325.095158] LustreError: Skipped 3 previous similar messages [ 325.343318] LustreError: 21325:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761322091 with bad export cookie 13026536000226558726 [ 325.345162] LustreError: 166-1: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 325.348324] LustreError: 21325:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 333.280172] Lustre: 21834:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761322093/real 1761322093] req@00000000509c95dd x1846879798229568/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761322099 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 341.187801] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 345.288197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 345.490516] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 345.523453] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 345.526284] Lustre: Skipped 2 previous similar messages [ 347.240984] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 350.648337] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 352.533890] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 355.937474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 357.758846] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 361.397384] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 361.424980] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 361.429594] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 361.492210] Lustre: lustre-OST0001: new disk, initializing [ 361.495423] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 363.207928] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 363.212314] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 363.423223] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 368.409239] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 369.884194] Lustre: Setting parameter general.lod.*.mdt_hash in log params [ 381.262343] Lustre: server umount lustre-MDT0000 complete [ 381.264742] Lustre: Skipped 2 previous similar messages [ 381.409072] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 381.413179] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 381.420830] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 381.426623] LustreError: Skipped 1 previous similar message [ 382.775217] LustreError: 24012:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761322149 with bad export cookie 13026536000226559573 [ 382.777953] LustreError: 166-1: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 382.779604] LustreError: 24012:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 392.672222] Lustre: 27030:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761322152/real 1761322152] req@000000005a03d309 x1846879798253504/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761322158 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 400.352264] Lustre: 27231:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761322160/real 1761322160] req@000000005a03d309 x1846879798254016/t0(0) o39->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761322166 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 406.608729] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 410.670426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 410.880439] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 410.883474] Lustre: Skipped 3 previous similar messages [ 412.556573] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 416.107664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 417.955933] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 421.566564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 423.518154] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 427.286750] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 429.073145] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 435.924245] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 437.164757] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 437.167290] Lustre: Skipped 1 previous similar message [ 446.948217] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 446.950751] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 446.952498] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 446.953570] Lustre: Skipped 5 previous similar messages [ 450.056437] Lustre: server umount lustre-MDT0000 complete [ 450.057940] Lustre: Skipped 3 previous similar messages [ 451.403577] LustreError: 28443:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761322217 with bad export cookie 13026536000226560819 [ 451.407519] LustreError: 166-1: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 451.408751] LustreError: 28443:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 459.232156] Lustre: 3334:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761322218/real 1761322218] req@000000002c7f1319 x1846879798272576/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761322225 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0' [ 459.244226] Lustre: 3334:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 464.135770] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 12:10:28 (1761322228) [ 475.180140] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 479.327658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 479.498495] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 479.506768] LustreError: Skipped 5 previous similar messages [ 481.254561] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 484.565497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 486.241828] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 489.514424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 491.302251] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 494.649092] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 496.428694] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 500.176758] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 501.604364] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 510.433782] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 510.435386] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 510.439494] Lustre: Skipped 3 previous similar messages [ 515.629876] LustreError: 34276:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761322281 with bad export cookie 13026536000226561890 [ 515.633900] LustreError: 166-1: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 515.635757] LustreError: 34276:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 523.744259] Lustre: 37849:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761322283/real 1761322283] req@00000000b2d9c606 x1846879798292160/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761322289 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 535.292939] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 12:11:40 (1761322300) [ 541.066149] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 544.761925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 544.952962] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 544.956259] Lustre: Skipped 7 previous similar messages [ 546.516285] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 549.815839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 551.501594] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 554.867564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 556.795405] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 560.277044] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 561.983914] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 565.391951] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 573.227045] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.27@tcp (stopping) [ 573.230278] Lustre: Skipped 7 previous similar messages [ 575.457817] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 575.461787] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 575.467266] Lustre: Skipped 2 previous similar messages [ 579.407855] Lustre: server umount lustre-MDT0000 complete [ 579.409725] Lustre: Skipped 7 previous similar messages [ 580.964131] LustreError: 39455:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761322347 with bad export cookie 13026536000226562961 [ 580.967225] LustreError: 166-1: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 580.968638] LustreError: 39455:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 591.002373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 598.497590] Lustre: Evicted from MGS (at 192.168.201.127@tcp) after server handle changed from 0xb4c78d7983579391 to 0xb4c78d798357980d [ 598.502812] Lustre: MGC192.168.201.127@tcp: Connection restored to (at 0@lo) [ 598.646298] BUG: spinlock bad magic on CPU#2, lctl/43386 [ 598.648504] lock: 0xffff90e3351d6900, .magic: 00000000, .owner: /-1, .owner_cpu: 0 [ 598.651876] CPU: 2 PID: 43386 Comm: lctl Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 [ 598.655496] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-2.fc40 04/01/2014 [ 598.657951] Call Trace: [ 598.659840] ? dump_stack+0xbb/0x10e [ 598.660904] ? spin_dump.cold.3+0x32/0x65 [ 598.662187] ? do_raw_spin_lock+0xce/0x190 [ 598.663414] ? _raw_spin_lock+0x21/0x30 [ 598.664745] ? mdd_changelog_mask_seq_write+0x61a/0x660 [mdd] [ 598.666560] ? __seccomp_filter+0x4b/0x680 [ 598.667983] ? full_proxy_write+0x5e/0xa0 [ 598.670073] ? __vfs_write+0x1c/0x60 [ 598.671164] ? vfs_write+0xd8/0x2b0 [ 598.672307] ? ksys_write+0x66/0x120 [ 598.673712] ? __x64_sys_write+0x1e/0x30 [ 598.675239] ? do_syscall_64+0xc1/0x440 [ 598.676236] ? entry_SYSCALL_64_after_hwframe+0x49/0xae [ 600.307793] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 603.590187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 604.773255] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to (at 0@lo) [ 605.467128] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 607.846278] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 607.848851] Lustre: Skipped 1 previous similar message [ 608.604837] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 610.302634] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 1 sec [ 612.521855] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 613.208251] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 629.216168] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 630.720310] LustreError: 40277:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761322396 with bad export cookie 13026536000226564109 [ 630.723275] LustreError: 166-1: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 630.725139] LustreError: 40277:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 638.432216] Lustre: 45600:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761322398/real 1761322398] req@00000000de7cf056 x1846879798327104/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761322404 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 638.441473] Lustre: 45600:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 649.335311] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 12:13:34 (1761322414) [ 654.725921] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 658.444489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 658.602759] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 658.608846] LustreError: Skipped 17 previous similar messages [ 660.336157] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 663.648037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 665.559128] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 668.890615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 670.625833] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 673.772687] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 675.581662] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 679.245798] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 680.830961] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 680.833992] Lustre: Skipped 2 previous similar messages [ 700.896162] Lustre: 50796:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761322461/real 1761322461] req@0000000009202152 x1846879798347712/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761322467 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 700.904022] Lustre: 50796:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 711.816968] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 12:14:36 (1761322476) [ 712.700990] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 713.314111] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 12:14:38 (1761322478) [ 717.219611] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [ 720.038370] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 723.558733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 725.689164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 728.312628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 728.344427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 728.453467] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 728.493459] Lustre: lustre-MDT0000: new disk, initializing [ 728.530558] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 730.228037] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 734.463877] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 737.038178] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 737.073168] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 737.077407] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 738.846688] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 743.005359] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 743.715873] Lustre: DEBUG MARKER: create test files [ 754.658484] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 754.658581] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 754.665817] Lustre: Skipped 13 previous similar messages [ 754.667993] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 754.669793] Lustre: Skipped 26 previous similar messages [ 766.944171] Lustre: 55479:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761322527/real 1761322527] req@0000000092f52637 x1846879798355200/t0(0) o251->MGC192.168.201.127@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761322533 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 766.956777] Lustre: 55479:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 772.072095] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 773.163129] Key type lgssc unregistered [ 773.281388] LNet: 56081:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 773.284447] LNet: Removed LNI 192.168.201.127@tcp [ 773.640139] Key type .llcrypt unregistered [ 773.641780] Key type ._llcrypt unregistered [ 784.857600] alg: No test for adler32 (adler32-zlib) [ 785.609376] Key type ._llcrypt registered [ 785.610900] Key type .llcrypt registered [ 785.648458] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 786.124635] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 786.273336] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 786.358545] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 786.360665] LNet: Accept secure, port 988 [ 787.952272] Key type lgssc registered [ 788.369678] Lustre: Echo OBD driver; http://www.lustre.org/ [ 791.258797] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 792.435401] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 794.026919] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 796.871345] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 799.469049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 799.630095] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 801.330098] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 804.172417] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 806.756497] Lustre: lustre-OST0000: deleting orphan objects from 0x0:102 to 0x0:129 [ 806.779228] Lustre: 58922:0:(osd_internal.h:1344:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 515 < left 544, rollback = 9 [ 806.783109] Lustre: 58922:0:(osd_handler.c:1947:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 806.786269] Lustre: 58922:0:(osd_handler.c:1954:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 806.789445] Lustre: 58922:0:(osd_handler.c:1964:osd_trans_dump_creds()) write: 1/1/1, punch: 0/0/0, quota 1/3/0 [ 806.792102] Lustre: 58922:0:(osd_handler.c:1971:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 806.794438] Lustre: 58922:0:(osd_handler.c:1978:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 812.667379] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 814.559543] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 817.121642] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 817.121680] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 817.126830] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 822.240979] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 822.701745] Lustre: server umount lustre-OST0000 complete [ 830.432163] Lustre: 59932:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761322590/real 1761322590] req@00000000e79b51b8 x1846880556291328/t0(0) o251->MGC192.168.201.127@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761322596 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 830.502104] Lustre: server umount lustre-MDT0000 complete [ 835.590497] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 836.693461] Key type lgssc unregistered [ 836.821420] LNet: 60535:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 836.825076] LNet: Removed LNI 192.168.201.127@tcp [ 837.157793] Key type .llcrypt unregistered [ 837.159312] Key type ._llcrypt unregistered [ 849.900876] alg: No test for adler32 (adler32-zlib) [ 850.652506] Key type ._llcrypt registered [ 850.654032] Key type .llcrypt registered [ 850.696453] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 851.148023] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 851.325432] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 851.416488] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 851.419341] LNet: Accept secure, port 988 [ 853.008162] Key type lgssc registered [ 853.427792] Lustre: Echo OBD driver; http://www.lustre.org/ [ 856.614091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 857.811394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 859.441025] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 862.375520] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 864.886671] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 864.998669] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 866.513192] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 869.310923] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 873.125407] Lustre: lustre-OST0000: deleting orphan objects from 0x0:140 to 0x0:161 [ 873.156859] Lustre: 63470:0:(osd_internal.h:1344:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 515 < left 544, rollback = 9 [ 873.161450] Lustre: 63470:0:(osd_handler.c:1947:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 873.165112] Lustre: 63470:0:(osd_handler.c:1954:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 873.169141] Lustre: 63470:0:(osd_handler.c:1964:osd_trans_dump_creds()) write: 1/1/1, punch: 0/0/0, quota 1/3/0 [ 873.172959] Lustre: 63470:0:(osd_handler.c:1971:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 873.176179] Lustre: 63470:0:(osd_handler.c:1978:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 877.803071] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 883.682523] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 883.682603] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 883.690661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 886.365644] Lustre: server umount lustre-OST0000 complete [ 892.896175] Lustre: 64435:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761322653/real 1761322653] req@00000000c39fe200 x1846880624448832/t0(0) o251->MGC192.168.201.127@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761322659 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 892.970946] Lustre: server umount lustre-MDT0000 complete [ 898.092276] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 899.208830] Key type lgssc unregistered [ 899.332672] LNet: 65039:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 899.336534] LNet: Removed LNI 192.168.201.127@tcp [ 899.661934] Key type .llcrypt unregistered [ 899.663494] Key type ._llcrypt unregistered [ 906.625983] alg: No test for adler32 (adler32-zlib) [ 907.377410] Key type ._llcrypt registered [ 907.379029] Key type .llcrypt registered [ 907.428455] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [ 913.972654] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 914.332463] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 914.556349] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 914.639546] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 914.642103] LNet: Accept secure, port 988 [ 916.224437] Key type lgssc registered [ 916.629952] Lustre: Echo OBD driver; http://www.lustre.org/ [ 920.210621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 922.787106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 925.102525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 927.524500] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 930.210790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 930.235189] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 931.334226] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 931.349057] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 931.388229] Lustre: lustre-MDT0000: new disk, initializing [ 931.422476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 931.431276] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 932.988209] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 937.513817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 937.544899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 937.585610] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 937.607079] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 937.609752] Lustre: Skipped 1 previous similar message [ 937.648300] Lustre: lustre-MDT0001: new disk, initializing [ 937.684133] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 937.698286] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 937.702966] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 939.223338] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 943.373450] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 945.450189] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 947.953807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 947.983835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 947.987170] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 948.087388] Lustre: lustre-OST0000: new disk, initializing [ 948.090026] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 948.119204] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 948.310428] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 948.315271] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 949.670538] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 953.760199] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 955.348820] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 955.426404] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 956.960236] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 957.041073] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 958.945840] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 958.946120] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 958.947181] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 958.948674] LustreError: Skipped 1 previous similar message [ 958.952794] Lustre: Skipped 1 previous similar message [ 964.066154] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 964.069159] Lustre: Skipped 2 previous similar messages [ 969.186309] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 969.188742] Lustre: Skipped 1 previous similar message [ 972.256218] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 972.332716] Lustre: server umount lustre-OST0000 complete [ 973.792907] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 973.796347] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 973.802165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 979.424950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 979.427134] Lustre: Skipped 3 previous similar messages [ 988.128184] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 988.167604] Lustre: server umount lustre-MDT0000 complete [ 989.452219] LustreError: 68303:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761322755 with bad export cookie 9972939185660006509 [ 989.454273] LustreError: 166-1: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 989.455524] LustreError: 68303:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 989.569146] Lustre: server umount lustre-MDT0001 complete [ 992.356084] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 12:19:17 (1761322757) [ 995.467702] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 999.903940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1008.964116] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [ 1011.864059] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 1015.219513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1017.458583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1019.450193] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1021.571587] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1023.914240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1023.938554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1024.014440] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1024.026364] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1024.061223] Lustre: lustre-MDT0000: new disk, initializing [ 1024.085300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1024.093253] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1025.509079] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 1029.786744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1029.806222] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1029.838383] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1029.856632] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1029.858950] Lustre: Skipped 1 previous similar message [ 1029.900869] Lustre: lustre-MDT0001: new disk, initializing [ 1029.937972] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1029.948989] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1029.952299] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1031.404374] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 1035.682585] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1038.504693] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1044.783848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1044.850338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1044.854180] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1045.138384] Lustre: lustre-OST0000: new disk, initializing [ 1045.141226] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1045.252114] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1049.483753] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 1054.272231] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1054.289888] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1058.091246] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1061.698701] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1062.185088] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1066.654516] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 1066.828690] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1070.053317] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1070.054077] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1070.062349] LustreError: Skipped 1 previous similar message [ 1070.064819] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1070.064828] Lustre: Skipped 2 previous similar messages [ 1070.078871] Lustre: Skipped 2 previous similar messages [ 1083.360139] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1083.621851] Lustre: server umount lustre-OST0000 complete [ 1090.529828] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1090.558378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1090.576243] Lustre: Skipped 6 previous similar messages [ 1102.304171] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1102.547163] Lustre: server umount lustre-MDT0000 complete [ 1105.887431] LustreError: 76593:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761322872 with bad export cookie 9972939185660007349 [ 1105.888442] LustreError: 166-1: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1105.909709] LustreError: 76593:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1109.137441] hrtimer: interrupt took 3505892 ns [ 1113.086710] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 12:21:17 (1761322877) [ 1117.389676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1117.725765] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1117.770664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1120.459612] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 1122.788925] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1125.825884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1128.295583] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 1133.180135] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1136.214036] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1141.241813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1141.288418] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1144.325525] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 1148.633341] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1150.817382] Lustre: 83886:0:(genops.c:1710:obd_export_evict_by_uuid()) MGS: evicting 9b5ffa96-dbce-4cd5-8f14-9243ef041ef7 at adminstrative request [ 1152.997928] LustreError: 166-1: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1153.007162] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 1159.138919] Lustre: MGS: Client 9b5ffa96-dbce-4cd5-8f14-9243ef041ef7 (at 0@lo) reconnecting [ 1159.151953] Lustre: Evicted from MGS (at 192.168.201.127@tcp) after server handle changed from 0x8a66ff59b7d696fd to 0x8a66ff59b7d6999d [ 1159.162478] Lustre: MGC192.168.201.127@tcp: Connection restored to (at 0@lo) [ 1185.558910] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1185.613411] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1185.617928] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 1185.697930] Lustre: lustre-OST0001: new disk, initializing [ 1185.702113] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 1185.759925] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 1185.764248] Lustre: Skipped 2 previous similar messages [ 1188.373534] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 1193.511415] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1193.516624] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1194.905319] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 1198.560951] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1198.562288] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1198.570148] LustreError: Skipped 1 previous similar message [ 1198.583546] Lustre: Skipped 1 previous similar message [ 1198.589377] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 1198.602578] Lustre: Skipped 5 previous similar messages [ 1203.024328] Lustre: server umount lustre-OST0001 complete [ 1203.029941] Lustre: Skipped 1 previous similar message [ 1203.682891] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1203.688966] LustreError: Skipped 1 previous similar message [ 1208.291205] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1208.294704] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1208.803189] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1208.810595] LustreError: Skipped 1 previous similar message [ 1218.533545] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1228.256254] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1228.327978] Lustre: server umount lustre-MDT0000 complete [ 1228.332553] Lustre: Skipped 1 previous similar message [ 1230.306106] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1230.325634] LustreError: Skipped 1 previous similar message [ 1231.079633] LustreError: 81127:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761322997 with bad export cookie 9972939185660008861 [ 1231.080317] LustreError: 166-1: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1231.089709] LustreError: 81127:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1237.499121] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 12:23:21 (1761323001) [ 1238.680085] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 1239.976520] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 12:23:24 (1761323004) [ 1241.074514] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 1242.331374] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 12:23:26 (1761323006) [ 1243.511555] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 1244.842619] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 12:23:29 (1761323009) [ 1246.006657] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 1249.025309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1249.628725] loop: module loaded [ 1249.635349] loop0: detected capacity change from 0 to 4294967296 [ 1249.669517] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 1250.642433] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 1252.368788] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 1254.780101] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 1260.044757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1262.715106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1265.209149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1267.466189] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1271.506839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1271.554455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1271.560621] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1271.586149] LustreError: 88329:0:(obd_config.c:774:class_setup()) setup lustre-OST0000-osd failed (-28) [ 1271.589893] LustreError: 88329:0:(obd_mount.c:200:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 1271.593723] LustreError: 88329:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 1271.597259] LustreError: 88329:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 1282.750930] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [ 1288.365217] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 1295.054667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1299.410535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1303.825125] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1307.660840] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1311.892589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1311.943866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1312.099511] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1312.120668] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1312.173857] Lustre: lustre-MDT0000: new disk, initializing [ 1312.231051] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1312.241104] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1314.816557] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 1321.898631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1321.952960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1322.017975] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1322.203372] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1324.997232] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 1331.729168] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1335.706534] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1339.904894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1339.946972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1339.953398] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1342.835863] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 1346.572236] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1349.169320] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1352.255060] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1352.437815] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1355.519894] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 1355.671409] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1361.892850] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1361.895175] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1361.897753] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1361.897759] Lustre: Skipped 10 previous similar messages [ 1361.919323] Lustre: Skipped 4 previous similar messages [ 1363.172183] Lustre: server umount lustre-OST0000 complete [ 1363.174539] Lustre: Skipped 1 previous similar message [ 1372.133811] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1372.140644] LustreError: Skipped 1 previous similar message [ 1374.172716] LustreError: 91185:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761323140 with bad export cookie 9972939185660009435 [ 1374.175651] LustreError: 166-1: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1374.182204] LustreError: 91185:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1379.404777] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 12:25:43 (1761323143) [ 1383.650342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1386.381742] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 1391.443100] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1394.252778] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 1398.945131] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1402.869656] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1407.223844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1410.110859] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 1415.329505] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1419.984036] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1420.023186] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1420.031340] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 1420.120778] Lustre: lustre-OST0001: new disk, initializing [ 1420.122868] Lustre: Skipped 2 previous similar messages [ 1420.125498] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 1420.131958] Lustre: Skipped 3 previous similar messages [ 1422.951092] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 1429.263467] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 1430.075236] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1430.079132] Lustre: Skipped 2 previous similar messages [ 1430.081471] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1442.757948] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1455.698263] Lustre: Failing over lustre-MDT0000 [ 1455.988212] Lustre: server umount lustre-MDT0000 complete [ 1455.990740] Lustre: Skipped 2 previous similar messages [ 1456.098639] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1456.102476] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1456.117085] Lustre: Skipped 4 previous similar messages [ 1456.129041] LustreError: Skipped 4 previous similar messages [ 1463.265735] Lustre: 66537:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761323222/real 1761323222] req@00000000495c2b69 x1846880690612160/t0(0) o400->MGC192.168.201.127@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761323229 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1463.281717] LustreError: 166-1: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1475.083077] LDISKFS-fs (dm-0): recovery complete [ 1475.090709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1480.684046] Lustre: Evicted from MGS (at 192.168.201.127@tcp) after server handle changed from 0x8a66ff59b7d69f31 to 0x8a66ff59b7d72d14 [ 1480.694419] Lustre: MGC192.168.201.127@tcp: Connection restored to (at 0@lo) [ 1480.875905] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-60 [ 1480.881389] Lustre: Skipped 6 previous similar messages [ 1480.920215] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1483.290956] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 1483.559446] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 1483.562996] Lustre: 95735:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 1486.306187] Lustre: 95740:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 1486.320300] Lustre: 95740:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 1 previous similar message [ 1486.326380] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 1486.366994] LustreError: 100894:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 1486.688539] LustreError: 100894:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 1487.018635] LustreError: 100894:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 1487.032387] LustreError: 100894:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 1 previous similar message [ 1487.344103] LustreError: 100894:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 1487.348318] LustreError: 100894:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 1 previous similar message [ 1487.352953] Lustre: 100894:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1487.360665] Lustre: 100894:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 7 previous similar messages [ 1488.309436] LustreError: 100894:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 1488.316240] LustreError: 100894:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 3 previous similar messages [ 1488.632151] LustreError: 100894:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 1488.635133] LustreError: 100894:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 3 previous similar messages [ 1489.579141] Lustre: 100894:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1489.586087] Lustre: 100894:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 6 previous similar messages [ 1490.531969] LustreError: 100894:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 1490.536704] LustreError: 100894:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 6 previous similar messages [ 1490.848124] LustreError: 100894:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 1490.853344] LustreError: 100894:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 6 previous similar messages [ 1493.688910] Lustre: 100894:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1493.693592] Lustre: 100894:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 12 previous similar messages [ 1494.643610] LustreError: 100894:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 1494.650241] LustreError: 100894:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 12 previous similar messages [ 1494.968616] LustreError: 100894:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 1494.974782] LustreError: 100894:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 12 previous similar messages [ 1501.880910] Lustre: 100894:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1501.886868] Lustre: 100894:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 25 previous similar messages [ 1502.850494] LustreError: 100894:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 1502.860282] LustreError: 100894:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 25 previous similar messages [ 1503.176271] LustreError: 100894:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 1503.182741] LustreError: 100894:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 25 previous similar messages [ 1518.004834] Lustre: 100894:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1518.020941] Lustre: 100894:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 50 previous similar messages [ 1518.969086] LustreError: 100894:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 1518.978039] LustreError: 100894:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 50 previous similar messages [ 1519.296200] LustreError: 100894:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 1519.302757] LustreError: 100894:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 50 previous similar messages [ 1543.192210] Lustre: 100894:0:(ldlm_lib.c:2278:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 1543.200679] Lustre: 100894:0:(ldlm_lib.c:2288:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1543.211631] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 1543.222520] LustreError: 100894:0:(tgt_grant.c:257:tgt_grant_sanity_check()) mdt_obd_disconnect: tot_granted 2097152 != fo_tot_granted 4194304 [ 1543.232728] Lustre: 100894:0:(ldlm_lib.c:1803:abort_req_replay_queue()) @@@ aborted: req@00000000389fecf7 x1846880684241024/t0(8589934777) o101->f2c2fad9-2393-4489-8fd0-59cc72093988@192.168.201.27@tcp:425/0 lens 592/0 e 0 to 0 dl 1761323315 ref 1 fl Complete:/4/ffffffff rc 0/-1 job:'' [ 1543.253640] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to (at 0@lo) [ 1543.258720] Lustre: Skipped 2 previous similar messages [ 1543.260346] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 1543.293223] Lustre: lustre-OST0001: deleting orphan objects from 0x0:47 to 0x0:545 [ 1543.293419] Lustre: lustre-OST0000: deleting orphan objects from 0x0:79 to 0x0:577 [ 1549.280766] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1549.284740] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1549.292236] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1549.295129] Lustre: Skipped 3 previous similar messages [ 1557.992495] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1558.002113] LustreError: Skipped 42 previous similar messages [ 1574.190632] LustreError: 95766:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761323340 with bad export cookie 9972939185660046612 [ 1574.193503] LustreError: 166-1: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1574.197716] LustreError: 95766:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1579.379671] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 12:29:03 (1761323343) [ 1584.133758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1584.138530] Lustre: *** cfs_fail_loc=197, val=0*** [ 1601.700521] LustreError: 102916:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.201.127@tcp: failed processing log, type 1: rc = -5 [ 1612.965296] LustreError: 102916:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.201.127@tcp: failed processing log, type 4: rc = -110 [ 1618.103636] Lustre: *** cfs_fail_loc=197, val=0*** [ 1618.109218] Lustre: Skipped 65 previous similar messages [ 1644.112705] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 1648.494937] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1649.862506] Lustre: server umount lustre-OST0000 complete [ 1649.864598] Lustre: Skipped 4 previous similar messages [ 1654.993801] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 12:30:19 (1761323419) [ 1671.567486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1674.238945] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 1680.281398] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1682.810687] LustreError: 104575:0:(osp_object.c:629:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1682.816177] LustreError: 104575:0:(lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1682.820160] LustreError: 104575:0:(lod_dev.c:424:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000 get update log failed: rc = -5 [ 1689.056160] Lustre: 105264:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761323449/real 1761323449] req@0000000065e5b13a x1846880690643456/t0(0) o251->MGC192.168.201.127@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761323455 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1698.496676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1708.831899] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [ 1713.441899] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 1718.320901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1721.362163] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1724.681111] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1728.289084] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1732.039599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1732.094278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1732.224606] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1732.244372] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1732.290763] Lustre: lustre-MDT0000: new disk, initializing [ 1732.373538] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1734.599196] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 1741.148230] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1741.186933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1741.371067] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1741.375449] Lustre: Skipped 3 previous similar messages [ 1741.399662] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1744.101181] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 1749.917414] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1753.221494] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1756.924630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1756.980195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1756.984994] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1759.658732] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 1765.158635] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1768.142458] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1768.269227] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1770.644748] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 1770.743899] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1773.539900] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1773.540728] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1773.546224] LustreError: Skipped 2 previous similar messages [ 1773.554725] Lustre: Skipped 6 previous similar messages [ 1785.824334] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1802.210338] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1804.074875] LustreError: 108885:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761323570 with bad export cookie 9972939185660057245 [ 1804.077315] LustreError: 166-1: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1804.083582] LustreError: 108885:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1808.206420] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 12:32:52 (1761323572) [ 1813.050147] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 1814.505711] Key type lgssc unregistered [ 1814.654536] LNet: 113570:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1814.660820] LNet: Removed LNI 192.168.201.127@tcp [ 1815.113575] Key type .llcrypt unregistered [ 1815.115321] Key type ._llcrypt unregistered [ 1824.498972] alg: No test for adler32 (adler32-zlib) [ 1825.251556] Key type ._llcrypt registered [ 1825.253380] Key type .llcrypt registered [ 1825.328403] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [ 1833.074881] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 1833.530988] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1833.727966] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 1833.839478] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 1833.843700] LNet: Accept secure, port 988 [ 1835.448298] Key type lgssc registered [ 1836.101304] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1841.707526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1845.818030] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1849.357040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1852.812635] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1855.456995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1858.943489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1858.996993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1860.170073] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1860.192408] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1860.260733] Lustre: lustre-MDT0000: new disk, initializing [ 1860.319290] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1860.329367] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1862.732491] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 1869.278405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1869.312718] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1869.317817] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1869.473777] Lustre: lustre-OST0000: new disk, initializing [ 1869.479671] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1869.483056] Lustre: Skipped 1 previous similar message [ 1869.494205] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 1869.545520] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1871.755789] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 1877.654972] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1877.705884] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1877.713042] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 1877.790513] Lustre: lustre-OST0001: new disk, initializing [ 1877.794666] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 1877.837587] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 1880.178796] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 1889.764652] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1889.766172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1889.778990] Lustre: Skipped 1 previous similar message [ 1891.969922] Lustre: server umount lustre-MDT0000 complete [ 1895.419436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1897.597218] LustreError: 117053:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761323663 with bad export cookie 14953767048812779825 [ 1897.598963] LustreError: 166-1: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1897.602787] LustreError: 117053:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1897.628177] Lustre: server umount lustre-OST0000 complete [ 1899.433394] Lustre: server umount lustre-OST0001 complete [ 1904.145534] Lustre: DEBUG MARKER: == conf-sanity test 88: check the default mount options can be overridden ========================================================== 12:34:28 (1761323668) [ 1908.743997] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 1913.111395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1913.563780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1916.266325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1916.802591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1920.012106] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 12:34:44 (1761323684) [ 1926.469835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1927.630626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1928.743544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1929.755918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1930.349533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1931.446193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1940.489854] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [ 1944.895053] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 1949.565877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1952.531319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1955.404064] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1958.408772] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1962.412687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1962.466967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1962.649124] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1962.681744] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1962.749667] Lustre: lustre-MDT0000: new disk, initializing [ 1962.823388] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1962.851722] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1965.287434] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 1971.864638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1971.914892] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1971.990453] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1972.027664] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1972.030900] Lustre: Skipped 1 previous similar message [ 1972.082719] Lustre: lustre-MDT0001: new disk, initializing [ 1972.141405] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1972.186601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1972.192441] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1974.342625] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 1979.827383] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1982.766529] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1986.105974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1986.143478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1986.147372] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1986.273388] Lustre: lustre-OST0000: new disk, initializing [ 1986.279563] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1986.325792] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1988.165061] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 1993.069321] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1995.129830] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1996.297839] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1996.302683] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1997.311361] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 1999.409352] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 1999.516468] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2001.377919] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2001.379682] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2001.387353] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2001.392572] Lustre: Skipped 1 previous similar message [ 2001.397408] Lustre: Skipped 2 previous similar messages [ 2002.493775] Lustre: server umount lustre-OST0000 complete [ 2006.500372] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2006.502120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2006.507899] Lustre: Skipped 1 previous similar message [ 2010.323023] Lustre: server umount lustre-MDT0000 complete [ 2011.617032] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2011.624246] LustreError: Skipped 1 previous similar message [ 2012.021438] LustreError: 125596:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761323778 with bad export cookie 14953767048812781008 [ 2012.028305] LustreError: 166-1: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2016.255561] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 12:36:20 (1761323780) [ 2019.379216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2019.573532] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2019.612221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2021.409924] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 2024.733533] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2026.638421] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 2029.763771] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2032.117084] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2034.763415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2036.687323] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 2039.902437] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2042.173130] Lustre: *** cfs_fail_loc=159, val=0*** [ 2049.331574] Lustre: lustre-MDT0000: Client bbc5cfb6-41a7-4acb-82bd-b849266a1432 (at 192.168.201.27@tcp) reconnecting [ 2049.766413] Lustre: *** cfs_fail_loc=159, val=0*** [ 2049.768782] Lustre: Skipped 5 previous similar messages [ 2057.010561] Lustre: lustre-MDT0000: Client bbc5cfb6-41a7-4acb-82bd-b849266a1432 (at 192.168.201.27@tcp) reconnecting [ 2060.773186] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2060.774088] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2060.776685] LustreError: Skipped 1 previous similar message [ 2060.782354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2060.786403] Lustre: Skipped 2 previous similar messages [ 2064.363949] Lustre: server umount lustre-OST0000 complete [ 2064.366045] Lustre: Skipped 1 previous similar message [ 2071.010584] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2071.010591] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2071.010596] Lustre: Skipped 1 previous similar message [ 2071.011894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2080.224188] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2082.132923] LustreError: 133011:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761323848 with bad export cookie 14953767048812781862 [ 2082.139496] LustreError: 166-1: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2082.151385] LustreError: 133011:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 2082.321032] Lustre: server umount lustre-MDT0001 complete [ 2082.322919] Lustre: Skipped 1 previous similar message [ 2088.027870] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 2089.425715] Key type lgssc unregistered [ 2089.570439] LNet: 134016:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2089.578045] LNet: Removed LNI 192.168.201.127@tcp [ 2089.943435] Key type .llcrypt unregistered [ 2089.945173] Key type ._llcrypt unregistered [ 2101.001436] alg: No test for adler32 (adler32-zlib) [ 2101.753477] Key type ._llcrypt registered [ 2101.754984] Key type .llcrypt registered [ 2101.795760] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 2102.174402] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2102.428336] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 2102.521590] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 2102.523887] LNet: Accept secure, port 988 [ 2104.120233] Key type lgssc registered [ 2104.617416] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2108.454291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2109.671675] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2109.705329] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2111.748929] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 2115.042281] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2115.794731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2115.972477] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2117.798080] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 2121.116638] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2123.659163] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2127.045563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2127.221392] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2129.121327] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 2132.441686] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2140.518397] Lustre: lustre-OST0000: deleting orphan objects from 0x0:42 to 0x0:65 [ 2147.906708] Lustre: *** cfs_fail_loc=159, val=0*** [ 2155.314712] Lustre: lustre-MDT0000: Client 562b5e87-1c2a-44fc-be53-eec1a475de1d (at 192.168.201.27@tcp) reconnecting [ 2156.065571] Lustre: *** cfs_fail_loc=159, val=0*** [ 2163.506323] Lustre: lustre-MDT0001: Client 562b5e87-1c2a-44fc-be53-eec1a475de1d (at 192.168.201.27@tcp) reconnecting [ 2163.920752] Lustre: *** cfs_fail_loc=159, val=0*** [ 2163.922836] Lustre: Skipped 4 previous similar messages [ 2171.191063] Lustre: lustre-MDT0001: Client 562b5e87-1c2a-44fc-be53-eec1a475de1d (at 192.168.201.27@tcp) reconnecting [ 2174.156711] Lustre: *** cfs_fail_loc=159, val=0*** [ 2174.158687] Lustre: Skipped 3 previous similar messages [ 2181.426422] Lustre: lustre-MDT0000: Client 9c5556c0-c338-4f24-84ca-d2593b5ff6eb (at 192.168.201.27@tcp) reconnecting [ 2181.911209] Lustre: *** cfs_fail_loc=159, val=0*** [ 2181.913033] Lustre: Skipped 14 previous similar messages [ 2189.112347] Lustre: lustre-MDT0000: Client 9c5556c0-c338-4f24-84ca-d2593b5ff6eb (at 192.168.201.27@tcp) reconnecting [ 2191.842859] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2191.844262] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2191.854258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2191.858593] Lustre: Skipped 1 previous similar message [ 2191.866523] Lustre: Skipped 1 previous similar message [ 2196.961720] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2196.966406] Lustre: Skipped 1 previous similar message [ 2197.617377] Lustre: server umount lustre-OST0000 complete [ 2202.083048] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2202.084374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2202.090722] Lustre: Skipped 1 previous similar message [ 2205.590346] Lustre: server umount lustre-MDT0000 complete [ 2207.203353] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2207.209579] LustreError: Skipped 1 previous similar message [ 2207.366551] LustreError: 135475:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761323973 with bad export cookie 3991857752930359430 [ 2207.370597] LustreError: 166-1: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2207.373594] LustreError: 135475:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2207.562671] Lustre: server umount lustre-MDT0001 complete [ 2212.915510] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 2214.086801] Key type lgssc unregistered [ 2214.208564] LNet: 139899:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2214.212703] LNet: Removed LNI 192.168.201.127@tcp [ 2214.549288] Key type .llcrypt unregistered [ 2214.550518] Key type ._llcrypt unregistered [ 2225.422358] alg: No test for adler32 (adler32-zlib) [ 2226.174401] Key type ._llcrypt registered [ 2226.177022] Key type .llcrypt registered [ 2226.231790] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 2226.787148] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2226.953336] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 2227.055959] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 2227.061134] LNet: Accept secure, port 988 [ 2228.664087] Key type lgssc registered [ 2229.243912] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2233.151370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2234.371309] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2234.393630] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2236.280737] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 2239.460170] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2240.261747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2240.446744] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2242.412860] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 2245.880728] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2248.395048] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2251.590174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2251.738956] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2253.554588] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 2254.823374] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:72 to 0x280000400:97 [ 2256.816825] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2264.043553] Lustre: lustre-OST0000: deleting orphan objects from 0x0:85 to 0x0:129 [ 2271.344692] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight in log params [ 2273.093591] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight in log params [ 2273.097434] Lustre: Skipped 1 previous similar message [ 2275.299407] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2275.304048] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2275.312699] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2277.436886] Lustre: server umount lustre-OST0000 complete [ 2279.394496] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2279.396803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2279.400807] Lustre: Skipped 1 previous similar message [ 2279.409381] Lustre: Skipped 1 previous similar message [ 2284.516772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2284.519381] Lustre: Skipped 1 previous similar message [ 2285.536234] Lustre: 144643:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761324045/real 1761324045] req@000000007bde1b3a x1846882067298944/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761324051 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2285.640107] Lustre: server umount lustre-MDT0000 complete [ 2287.461230] LustreError: 143315:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761324053 with bad export cookie 9405376100553788052 [ 2287.462576] LustreError: 166-1: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2287.467797] LustreError: 143315:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2287.667314] Lustre: server umount lustre-MDT0001 complete [ 2292.943546] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 2294.190876] Key type lgssc unregistered [ 2294.332474] LNet: 145451:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2294.337614] LNet: Removed LNI 192.168.201.127@tcp [ 2294.707315] Key type .llcrypt unregistered [ 2294.709240] Key type ._llcrypt unregistered [ 2305.791775] alg: No test for adler32 (adler32-zlib) [ 2306.544454] Key type ._llcrypt registered [ 2306.546073] Key type .llcrypt registered [ 2306.607879] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 2307.055771] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2307.295231] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 2307.409825] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 2307.412504] LNet: Accept secure, port 988 [ 2309.008170] Key type lgssc registered [ 2309.508585] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2312.991749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2314.173606] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2314.191300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2315.859464] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 2319.329049] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2319.576000] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2319.729195] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2321.526543] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 2324.336838] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2326.497148] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2329.074778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2329.187638] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2330.718068] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 2333.791122] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2335.338644] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:72 to 0x280000400:129 [ 2344.552712] Lustre: lustre-OST0000: deleting orphan objects from 0x0:85 to 0x0:161 [ 2351.827789] Lustre: *** cfs_fail_loc=159, val=0*** [ 2359.087770] Lustre: lustre-MDT0000: Client 904c2c1f-7aa6-4385-a89f-63e1e3fef273 (at 192.168.201.27@tcp) reconnecting [ 2360.291755] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2360.292036] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2360.292994] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2360.295576] Lustre: Skipped 1 previous similar message [ 2365.410246] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2365.413185] Lustre: Skipped 2 previous similar messages [ 2366.374920] Lustre: server umount lustre-OST0000 complete [ 2370.529876] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2370.533284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2370.534841] Lustre: Skipped 1 previous similar message [ 2375.649658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2375.652720] Lustre: Skipped 2 previous similar messages [ 2380.771515] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2380.775372] Lustre: Skipped 1 previous similar message [ 2382.304148] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2382.349605] Lustre: server umount lustre-MDT0000 complete [ 2383.918234] LustreError: 146893:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761324150 with bad export cookie 7381218054984745199 [ 2383.919628] LustreError: 166-1: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2383.922844] LustreError: 146893:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2384.070222] Lustre: server umount lustre-MDT0001 complete [ 2389.162357] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 2390.245162] Key type lgssc unregistered [ 2390.364610] LNet: 150696:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2390.368053] LNet: Removed LNI 192.168.201.127@tcp [ 2390.692743] Key type .llcrypt unregistered [ 2390.694764] Key type ._llcrypt unregistered [ 2401.212626] alg: No test for adler32 (adler32-zlib) [ 2401.964373] Key type ._llcrypt registered [ 2401.965809] Key type .llcrypt registered [ 2402.013466] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 2402.489578] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2402.684926] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 2402.796692] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 2402.799545] LNet: Accept secure, port 988 [ 2404.392154] Key type lgssc registered [ 2404.817153] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2408.055458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2409.246169] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2409.263825] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2410.933956] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 2414.405197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2414.586747] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2416.251812] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 2419.328862] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2421.763103] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2424.612755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2424.741954] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2426.463068] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 2428.841515] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:72 to 0x280000400:161 [ 2429.569330] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2439.083876] Lustre: lustre-OST0000: deleting orphan objects from 0x0:170 to 0x0:193 [ 2453.088734] Lustre: lustre-OST0000: evicting b54202a9-5314-417f-92f8-5c53a9964118 (at 192.168.201.27@tcp) by administrative request [ 2453.092591] Lustre: Skipped 1 previous similar message [ 2480.099306] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2480.099529] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2480.109614] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2480.112318] Lustre: Skipped 1 previous similar message [ 2485.219421] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2485.222302] Lustre: Skipped 1 previous similar message [ 2490.338426] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2490.340580] Lustre: Skipped 1 previous similar message [ 2491.872180] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2491.953750] Lustre: server umount lustre-OST0000 complete [ 2495.457474] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2495.458614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2495.463420] Lustre: Skipped 2 previous similar messages [ 2499.530540] Lustre: server umount lustre-MDT0000 complete [ 2500.578243] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2500.582487] LustreError: Skipped 1 previous similar message [ 2500.899013] LustreError: 152139:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761324267 with bad export cookie 332435194431627040 [ 2500.901661] LustreError: 166-1: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2500.903681] LustreError: 152139:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2501.044585] Lustre: server umount lustre-MDT0001 complete [ 2506.126247] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 2507.119452] Key type lgssc unregistered [ 2507.231336] LNet: 156372:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2507.233756] LNet: Removed LNI 192.168.201.127@tcp [ 2507.529575] Key type .llcrypt unregistered [ 2507.531388] Key type ._llcrypt unregistered [ 2534.335437] alg: No test for adler32 (adler32-zlib) [ 2535.087324] Key type ._llcrypt registered [ 2535.088904] Key type .llcrypt registered [ 2535.125801] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 2535.517348] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2535.735266] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 2535.812711] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 2535.815132] LNet: Accept secure, port 988 [ 2537.400161] Key type lgssc registered [ 2537.780108] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2540.654735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2541.808929] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2541.826126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2543.181253] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 2546.257659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2546.402793] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2548.019401] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 2550.747731] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2552.785799] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2555.178264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2555.311763] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2556.803719] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 2557.350418] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:225 [ 2559.498285] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2561.448075] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:72 to 0x280000400:193 [ 2567.649210] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2567.652775] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2567.657736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2571.746095] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2571.746500] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2571.747499] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2571.753680] Lustre: Skipped 1 previous similar message [ 2573.212465] Lustre: server umount lustre-OST0000 complete [ 2576.865844] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2576.866870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2576.870785] Lustre: Skipped 1 previous similar message [ 2580.549151] Lustre: server umount lustre-MDT0000 complete [ 2581.938460] LustreError: 158885:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761324348 with bad export cookie 13756805651925908072 [ 2581.941123] LustreError: 166-1: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2581.943204] LustreError: 158885:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2582.080798] Lustre: server umount lustre-MDT0001 complete [ 2586.922030] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 2588.088504] Key type lgssc unregistered [ 2588.194429] LNet: 162569:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2588.197416] LNet: Removed LNI 192.168.201.127@tcp [ 2588.496174] Key type .llcrypt unregistered [ 2588.497874] Key type ._llcrypt unregistered [ 2598.085831] alg: No test for adler32 (adler32-zlib) [ 2598.837472] Key type ._llcrypt registered [ 2598.839022] Key type .llcrypt registered [ 2598.883896] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 2599.313079] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2599.472968] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 2599.557328] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 2599.559376] LNet: Accept secure, port 988 [ 2601.136301] Key type lgssc registered [ 2601.575155] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2604.133106] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2606.780451] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 12:46:11 (1761324371) [ 2612.217246] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [ 2615.214231] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 2618.761705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2620.943240] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2622.943157] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2625.122531] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2627.594969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2627.620353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2628.730062] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2628.741924] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2628.779152] Lustre: lustre-MDT0000: new disk, initializing [ 2628.811929] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2628.821627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2630.394385] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 2634.826243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2634.853631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2634.890331] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2634.908713] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 2634.911022] Lustre: Skipped 1 previous similar message [ 2634.945096] Lustre: lustre-MDT0001: new disk, initializing [ 2634.981817] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2634.998208] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2635.003584] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2636.446139] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 2640.454930] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2642.595198] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2644.949661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2644.971628] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2644.975322] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2645.067466] Lustre: lustre-OST0000: new disk, initializing [ 2645.069374] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2645.095933] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2646.676917] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 2650.803532] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2651.638556] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2651.643079] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2661.857560] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2661.857898] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2661.861926] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2661.867013] Lustre: Skipped 1 previous similar message [ 2661.871849] Lustre: Skipped 1 previous similar message [ 2665.379820] Lustre: server umount lustre-OST0000 complete [ 2666.977104] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2666.977509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2666.980959] Lustre: Skipped 1 previous similar message [ 2672.097081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2672.100070] Lustre: Skipped 2 previous similar messages [ 2672.766249] Lustre: server umount lustre-MDT0000 complete [ 2674.022090] LustreError: 166959:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761324440 with bad export cookie 10258955269221149848 [ 2674.023871] LustreError: 166-1: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2674.027935] LustreError: 166959:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2674.165760] Lustre: server umount lustre-MDT0001 complete [ 2679.651441] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 2680.769976] Key type lgssc unregistered [ 2680.891329] LNet: 170962:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2680.895315] LNet: Removed LNI 192.168.201.127@tcp [ 2681.194082] Key type .llcrypt unregistered [ 2681.196181] Key type ._llcrypt unregistered [ 2691.844476] alg: No test for adler32 (adler32-zlib) [ 2692.596416] Key type ._llcrypt registered [ 2692.598156] Key type .llcrypt registered [ 2692.637117] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 2693.062642] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2693.219416] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 2693.302828] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 2693.304959] LNet: Accept secure, port 988 [ 2694.880127] Key type lgssc registered [ 2695.264988] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2698.233089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2699.410084] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2699.426151] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2700.989478] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 2704.337329] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2704.464304] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2706.051926] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 2709.033960] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2711.316752] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2713.546659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2713.680273] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2714.726949] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 2715.069950] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 2717.709485] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2720.675432] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2720.681785] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2720.688096] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2720.691772] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2720.696723] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.201.127@tcp (at 0@lo) [ 2720.697994] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:65 [ 2721.039396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2721.044827] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:97 [ 2721.386479] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2721.389239] Lustre: Skipped 1 previous similar message [ 2721.395281] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2721.400956] Lustre: Skipped 1 previous similar message [ 2721.403158] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2721.406284] Lustre: Skipped 1 previous similar message [ 2721.408576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2721.414451] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.201.127@tcp (at 0@lo) [ 2721.414701] Lustre: lustre-OST0000: deleting orphan objects from 0x0:159 to 0x0:193 [ 2721.417854] Lustre: Skipped 1 previous similar message [ 2721.743780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2721.750421] Lustre: lustre-OST0000: deleting orphan objects from 0x0:159 to 0x0:225 [ 2722.080958] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2722.086850] Lustre: lustre-OST0000: deleting orphan objects from 0x0:159 to 0x0:257 [ 2722.423540] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2722.426435] Lustre: Skipped 2 previous similar messages [ 2722.434619] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2722.439544] Lustre: Skipped 2 previous similar messages [ 2722.442099] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2722.444187] Lustre: Skipped 2 previous similar messages [ 2722.445490] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2722.450155] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.201.127@tcp (at 0@lo) [ 2722.450393] Lustre: lustre-OST0000: deleting orphan objects from 0x0:159 to 0x0:289 [ 2722.452598] Lustre: Skipped 2 previous similar messages [ 2722.821074] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2722.825255] Lustre: lustre-OST0000: deleting orphan objects from 0x0:159 to 0x0:321 [ 2723.171155] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2723.175898] Lustre: lustre-OST0000: deleting orphan objects from 0x0:159 to 0x0:353 [ 2723.505949] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2723.511307] Lustre: lustre-OST0000: deleting orphan objects from 0x0:159 to 0x0:385 [ 2723.848009] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2723.853825] Lustre: lustre-OST0000: deleting orphan objects from 0x0:159 to 0x0:417 [ 2728.928682] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2728.931425] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2728.934860] Lustre: Skipped 4 previous similar messages [ 2728.936457] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2729.441654] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2729.442446] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2732.646772] Lustre: server umount lustre-OST0000 complete [ 2734.561163] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2734.561950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2734.565919] Lustre: Skipped 2 previous similar messages [ 2734.570579] Lustre: Skipped 2 previous similar messages [ 2739.680981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2739.683671] Lustre: Skipped 1 previous similar message [ 2744.801796] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2744.804655] Lustre: Skipped 1 previous similar message [ 2748.384122] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2748.424357] Lustre: server umount lustre-MDT0000 complete [ 2749.691107] LustreError: 172404:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761324515 with bad export cookie 16975825041011396334 [ 2749.693540] LustreError: 166-1: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2749.694498] LustreError: 172404:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2749.819131] Lustre: server umount lustre-MDT0001 complete [ 2754.894429] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 2755.871798] Key type lgssc unregistered [ 2755.989282] LNet: 176878:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2755.991769] LNet: Removed LNI 192.168.201.127@tcp [ 2756.280164] Key type .llcrypt unregistered [ 2756.281789] Key type ._llcrypt unregistered [ 2765.492903] alg: No test for adler32 (adler32-zlib) [ 2766.245397] Key type ._llcrypt registered [ 2766.247011] Key type .llcrypt registered [ 2766.284847] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 2766.698338] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2766.845501] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 2766.935072] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 2766.937677] LNet: Accept secure, port 988 [ 2768.520128] Key type lgssc registered [ 2768.868447] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2771.440015] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2772.579096] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2772.591483] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2774.010690] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 2776.830530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2776.936734] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2778.360590] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 2781.027563] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2783.148347] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2785.565679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2785.675952] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2787.143025] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 2787.750106] Lustre: lustre-OST0000: deleting orphan objects from 0x0:159 to 0x0:449 [ 2789.773381] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2796.693924] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2797.299994] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2797.306141] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2797.308113] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2797.310925] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.201.127@tcp (at 0@lo) [ 2797.310937] LustreError: 178359:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 2801.447172] Lustre: lustre-OST0000: Not available for connect from 192.168.201.27@tcp (stopping) [ 2802.146079] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2802.146395] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2802.152459] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2802.153703] Lustre: Skipped 1 previous similar message [ 2806.561410] Lustre: lustre-OST0000: Not available for connect from 192.168.201.27@tcp (stopping) [ 2811.681463] Lustre: lustre-OST0000: Not available for connect from 192.168.201.27@tcp (stopping) [ 2811.685439] Lustre: Skipped 2 previous similar messages [ 2812.384139] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2812.459134] Lustre: server umount lustre-OST0000 complete [ 2817.384114] LustreError: 178359:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 2107 awake [ 2817.386581] LustreError: 178359:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -11 [ 2840.635100] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2840.774276] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2842.242300] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 2844.891736] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2863.270536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2863.276384] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2863.276392] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.201.127@tcp (at 0@lo) [ 2863.278394] Lustre: lustre-OST0000: deleting orphan objects from 0x0:159 to 0x0:481 [ 2863.284809] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:66 to 0x280000400:97 [ 2863.286274] Lustre: Skipped 1 previous similar message [ 2864.186604] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 18 sec [ 2866.270633] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9e24865c7800.ost_server_uuid 40 [ 2866.812512] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9e24865c7800.ost_server_uuid in FULL state after 0 sec [ 2868.707312] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2868.707798] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2868.709681] LustreError: Skipped 1 previous similar message [ 2868.714251] Lustre: Skipped 1 previous similar message [ 2868.716277] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2868.719895] Lustre: Skipped 3 previous similar messages [ 2873.945702] Lustre: server umount lustre-OST0000 complete [ 2878.945220] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2878.945838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2878.949010] Lustre: Skipped 1 previous similar message [ 2878.950759] Lustre: Skipped 2 previous similar messages [ 2881.288670] Lustre: server umount lustre-MDT0000 complete [ 2882.553804] LustreError: 178319:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761324648 with bad export cookie 4549972422655995040 [ 2882.555956] LustreError: 166-1: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2882.556905] LustreError: 178319:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2887.603266] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 2888.625544] Key type lgssc unregistered [ 2888.725380] LNet: 183649:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2888.727902] LNet: Removed LNI 192.168.201.127@tcp [ 2889.023177] Key type .llcrypt unregistered [ 2889.024062] Key type ._llcrypt unregistered [ 2900.164741] alg: No test for adler32 (adler32-zlib) [ 2900.916299] Key type ._llcrypt registered [ 2900.917273] Key type .llcrypt registered [ 2900.957166] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 2901.387575] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2901.629243] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 2901.704388] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 2901.705796] LNet: Accept secure, port 988 [ 2903.288149] Key type lgssc registered [ 2903.673516] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2906.715076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2907.851678] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2907.865192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2909.295323] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 2912.169659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2912.285568] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2913.724596] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 2916.604871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2916.720054] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2918.181326] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 2921.185830] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2921.210552] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2921.213103] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 2921.262454] Lustre: lustre-OST0001: new disk, initializing [ 2921.264403] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 2921.289844] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2922.840685] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 2923.372548] Lustre: lustre-OST0000: deleting orphan objects from 0x0:483 to 0x0:513 [ 2923.769587] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2923.773716] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2928.462200] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2931.210179] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 2944.482855] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2944.483789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2944.487800] Lustre: Skipped 2 previous similar messages [ 2944.491299] Lustre: Skipped 3 previous similar messages [ 2949.508339] Lustre: server umount lustre-MDT0000 complete [ 2949.601474] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2949.604698] LustreError: Skipped 3 previous similar messages [ 2950.746357] LustreError: 186210:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761324717 with bad export cookie 961148554391195752 [ 2950.749729] LustreError: 166-1: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2950.751362] LustreError: 186210:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2950.877739] Lustre: server umount lustre-MDT0001 complete [ 2958.304359] Lustre: 189211:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761324718/real 1761324718] req@00000000fd6ab6d1 x1846882774065792/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761324724 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 2958.336335] Lustre: server umount lustre-OST0000 complete [ 2964.960223] Lustre: 189412:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761324725/real 1761324725] req@00000000ac75b909 x1846882774066240/t0(0) o39->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761324731 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 2965.041638] Lustre: server umount lustre-OST0001 complete [ 2966.072851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 2966.101988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2966.472243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 2966.512733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2966.895419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 2966.915888] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2967.278542] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 2967.318413] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2972.006272] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 2975.294897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2975.436716] LustreError: 137-5: mylustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2975.455380] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2976.869850] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 2979.841896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2981.494611] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 2984.363698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2984.463692] Lustre: mylustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2984.467437] Lustre: Skipped 1 previous similar message [ 2985.509455] LustreError: 137-5: mylustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2985.916193] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 2988.961973] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2990.695507] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 2991.342258] Lustre: mylustre-OST0000: deleting orphan objects from 0x0:515 to 0x0:545 [ 2991.343468] Lustre: mylustre-OST0001: deleting orphan objects from 0x0:3 to 0x0:33 [ 2994.048127] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2995.391310] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 3005.921113] LustreError: 11-0: mylustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3005.924788] Lustre: mylustre-MDT0000-osp-MDT0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3005.929688] Lustre: Skipped 1 previous similar message [ 3005.932105] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3006.945228] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3006.946700] Lustre: mylustre-MDT0000-lwp-OST0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3006.951387] Lustre: Skipped 2 previous similar messages [ 3010.822498] Lustre: server umount mylustre-MDT0000 complete [ 3012.065865] LustreError: 137-5: mylustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3012.071228] LustreError: Skipped 3 previous similar messages [ 3012.092366] LustreError: 191670:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761324778 with bad export cookie 961148554391198461 [ 3012.096780] LustreError: 166-1: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3012.098240] LustreError: 191670:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3019.744179] Lustre: 194568:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761324779/real 1761324779] req@000000000c3ba3d8 x1846882774089856/t0(0) o39->mylustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761324785 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 3019.774175] Lustre: server umount mylustre-OST0000 complete [ 3019.776138] Lustre: Skipped 1 previous similar message [ 3027.424129] Lustre: 194769:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761324787/real 1761324787] req@0000000058610ce9 x1846882774090240/t0(0) o39->mylustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761324793 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 3028.637399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 3028.668730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3029.048883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 3029.072848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3029.444628] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 3029.477474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3029.852155] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 3029.885466] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3034.730176] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 3038.437515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3038.603871] LustreError: 137-5: tfs-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3038.626659] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3038.628841] Lustre: Skipped 1 previous similar message [ 3040.186897] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 3043.253150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3045.029863] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 3047.828061] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3049.289975] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 3052.091286] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3053.492492] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 3054.184814] Lustre: tfs-OST0000: deleting orphan objects from 0x0:515 to 0x0:577 [ 3054.190102] Lustre: tfs-OST0001: deleting orphan objects from 0x0:35 to 0x0:65 [ 3056.625399] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3057.729949] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 3064.802760] Lustre: tfs-MDT0000-lwp-OST0000: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3064.803778] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 3064.807475] Lustre: Skipped 2 previous similar messages [ 3064.809695] Lustre: Skipped 4 previous similar messages [ 3069.922273] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 3069.924822] Lustre: Skipped 5 previous similar messages [ 3070.606726] Lustre: server umount tfs-MDT0000 complete [ 3070.608706] Lustre: Skipped 1 previous similar message [ 3071.955126] LustreError: 197027:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761324838 with bad export cookie 961148554391200295 [ 3071.958295] LustreError: 166-1: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3071.961021] LustreError: 197027:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3079.648204] Lustre: 199864:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761324839/real 1761324839] req@0000000085bad4b8 x1846882774108352/t0(0) o39->tfs-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761324845 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 3088.546359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 3088.590786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3089.001705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 3089.034634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3089.414395] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 3089.437243] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3089.800219] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 3089.825727] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3094.993952] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 3098.617252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3098.772735] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3098.780403] LustreError: Skipped 1 previous similar message [ 3098.793466] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3098.795885] Lustre: Skipped 3 previous similar messages [ 3100.348146] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 3103.520273] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3105.092896] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 3107.876111] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3109.416475] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 3112.210903] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3113.552440] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 3114.538710] Lustre: lustre-OST0001: deleting orphan objects from 0x0:35 to 0x0:97 [ 3114.539044] Lustre: lustre-OST0000: deleting orphan objects from 0x0:515 to 0x0:609 [ 3121.736449] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3122.821534] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 3130.100529] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 12:54:54 (1761324894) [ 3144.672788] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3144.676712] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3144.682341] Lustre: Skipped 1 previous similar message [ 3144.684736] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3148.045839] Lustre: server umount lustre-MDT0000 complete [ 3148.047247] Lustre: Skipped 3 previous similar messages [ 3149.299035] LustreError: 204508:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761324915 with bad export cookie 961148554391201926 [ 3149.301385] LustreError: 166-1: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3149.303163] LustreError: 204508:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3150.304818] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3150.310964] LustreError: Skipped 2 previous similar messages [ 3171.296211] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3180.799902] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [ 3183.501111] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 3186.878633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3188.947885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3190.855240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3192.844905] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3195.035664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3195.055976] LDISKFS-fs (dm-0): Mount option "noacl" will be removed by 3.5 [ 3195.055976] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 3195.055976] [ 3195.062475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 3195.138982] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3195.150813] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3195.184606] Lustre: lustre-MDT0000: new disk, initializing [ 3195.207237] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3195.210349] Lustre: Skipped 3 previous similar messages [ 3195.217380] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3196.622154] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 3200.709439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3200.735111] LDISKFS-fs (dm-1): Mount option "noacl" will be removed by 3.5 [ 3200.735111] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 3200.735111] [ 3200.745108] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 3200.792063] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3200.794605] Lustre: Skipped 1 previous similar message [ 3200.830792] Lustre: lustre-MDT0001: new disk, initializing [ 3200.880321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3200.884683] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3202.284196] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 3206.412847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3206.432709] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3206.434836] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3206.528519] Lustre: lustre-OST0000: new disk, initializing [ 3206.530966] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3207.736942] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3207.739620] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3207.937949] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 3212.128831] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3212.154837] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3212.158613] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 3212.211678] Lustre: lustre-OST0001: new disk, initializing [ 3212.213621] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 3213.444688] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3213.447489] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3213.753076] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 3218.183135] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 12:56:23 (1761324983) [ 3221.631231] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 12:56:26 (1761324986) [ 3233.249072] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3233.249086] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3233.251630] LustreError: Skipped 1 previous similar message [ 3233.256714] Lustre: Skipped 5 previous similar messages [ 3233.257237] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3233.263682] Lustre: Skipped 11 previous similar messages [ 3239.092283] Lustre: server umount lustre-OST0000 complete [ 3239.093620] Lustre: Skipped 3 previous similar messages [ 3239.398538] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3239.403132] LustreError: Skipped 5 previous similar messages [ 3241.952696] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3247.911890] LustreError: 209478:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761325014 with bad export cookie 961148554391203445 [ 3247.914175] LustreError: 166-1: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3247.916368] LustreError: 209478:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3249.633598] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3249.636981] Lustre: Skipped 4 previous similar messages [ 3258.989846] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 3277.280132] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3279.769919] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [ 3285.559438] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 3286.620587] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3289.173229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3291.352745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3293.416676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3295.440557] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3297.688020] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3297.714716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3297.799302] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3297.802284] Lustre: Skipped 1 previous similar message [ 3297.812379] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3297.846225] Lustre: lustre-MDT0000: new disk, initializing [ 3297.871691] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3299.234660] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 3303.222307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3303.247809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3303.360386] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3304.766861] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 3308.604245] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3310.589655] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3312.901300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3312.927654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3312.931852] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3314.462978] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 3314.807772] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3314.810667] Lustre: Skipped 1 previous similar message [ 3314.812361] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3318.228531] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3325.409859] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3325.410406] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3325.412630] LustreError: Skipped 1 previous similar message [ 3325.416989] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3325.420163] Lustre: Skipped 9 previous similar messages [ 3342.304186] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3343.618889] LustreError: 218015:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761325109 with bad export cookie 961148554391204873 [ 3343.620748] LustreError: 166-1: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3343.624184] LustreError: 218015:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3348.565374] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 3349.633708] Key type lgssc unregistered [ 3349.759500] LNet: 218818:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3349.763168] LNet: Removed LNI 192.168.201.127@tcp [ 3350.027572] Key type .llcrypt unregistered [ 3350.029076] Key type ._llcrypt unregistered [ 3359.703354] alg: No test for adler32 (adler32-zlib) [ 3360.454384] Key type ._llcrypt registered [ 3360.456129] Key type .llcrypt registered [ 3360.497215] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 3360.882976] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3361.039053] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 3361.106962] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 3361.109521] LNet: Accept secure, port 988 [ 3362.688083] Key type lgssc registered [ 3363.001122] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3365.627608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3366.821522] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3366.838870] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3368.479162] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 3371.599629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3371.722865] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3373.302233] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 3376.062722] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3378.168641] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3380.595462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3380.754200] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3382.323167] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 3382.821407] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 3385.135550] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3385.524538] Lustre: Setting parameter lustre-OST0000.ost.unknown_param in log lustre-OST0000 [ 3387.874846] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3387.878613] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3387.878853] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3388.898019] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3392.362221] Lustre: server umount lustre-OST0000 complete [ 3394.018283] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3394.018854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3394.021827] Lustre: Skipped 1 previous similar message [ 3394.027043] Lustre: Skipped 2 previous similar messages [ 3399.137070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3399.139524] Lustre: Skipped 1 previous similar message [ 3399.744201] Lustre: server umount lustre-MDT0000 complete [ 3401.007208] LustreError: 220308:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761325167 with bad export cookie 1301492957974267113 [ 3401.009889] LustreError: 166-1: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3401.012030] LustreError: 220308:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3401.153234] Lustre: server umount lustre-MDT0001 complete [ 3404.692591] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 3405.810423] Key type lgssc unregistered [ 3405.922456] LNet: 224026:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3405.926559] LNet: Removed LNI 192.168.201.127@tcp [ 3406.216509] Key type .llcrypt unregistered [ 3406.218136] Key type ._llcrypt unregistered [ 3413.127862] alg: No test for adler32 (adler32-zlib) [ 3413.879433] Key type ._llcrypt registered [ 3413.881071] Key type .llcrypt registered [ 3413.920748] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 3414.461401] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3414.607074] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 3414.684586] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 3414.686030] LNet: Accept secure, port 988 [ 3416.256107] Key type lgssc registered [ 3416.631435] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3419.619056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3437.152369] LustreError: 225267:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.201.127@tcp: failed processing log, type 1: rc = -5 [ 3468.896385] LustreError: 225267:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.201.127@tcp: failed processing log, type 4: rc = -110 [ 3498.592224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3500.071581] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 3502.800456] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3503.628851] Lustre: server umount lustre-OST0000 complete [ 3509.126304] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 3510.221273] Key type lgssc unregistered [ 3510.356510] LNet: 226922:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3510.360099] LNet: Removed LNI 192.168.201.127@tcp [ 3510.691402] Key type .llcrypt unregistered [ 3510.692985] Key type ._llcrypt unregistered [ 3523.939757] alg: No test for adler32 (adler32-zlib) [ 3524.691406] Key type ._llcrypt registered [ 3524.692590] Key type .llcrypt registered [ 3524.731765] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 3525.303547] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3525.479742] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 3525.569611] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 3525.571108] LNet: Accept secure, port 988 [ 3527.160124] Key type lgssc registered [ 3527.525617] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3532.018405] loop0: detected capacity change from 0 to 204800000 [ 3532.075137] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3532.078544] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3532.229788] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3532.234190] blk_update_request: operation not supported error, dev loop0, sector 160240 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3532.256299] blk_update_request: operation not supported error, dev loop0, sector 160344 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3532.280426] blk_update_request: operation not supported error, dev loop0, sector 168552 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3532.296380] blk_update_request: operation not supported error, dev loop0, sector 173280 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3532.325628] blk_update_request: operation not supported error, dev loop0, sector 176912 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3532.363054] blk_update_request: operation not supported error, dev loop0, sector 185112 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3532.392418] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3532.767556] loop0: detected capacity change from 0 to 204800000 [ 3532.822915] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3533.118283] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3533.492916] loop0: detected capacity change from 0 to 204800000 [ 3533.671423] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3534.044791] loop0: detected capacity change from 0 to 204800000 [ 3534.219323] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3534.581621] loop0: detected capacity change from 0 to 204800000 [ 3534.597276] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3536.017575] loop0: detected capacity change from 0 to 204800000 [ 3536.028657] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3537.367275] loop0: detected capacity change from 0 to 204800000 [ 3537.377820] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3538.928736] loop0: detected capacity change from 0 to 204800000 [ 3538.936482] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3540.444579] loop0: detected capacity change from 0 to 204800000 [ 3540.480693] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3540.509678] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3540.536606] Lustre: 230377:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 3540.541509] Lustre: 230377:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 3540.545542] Lustre: 230377:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 3540.549797] Lustre: 230377:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 3540.553792] Lustre: 230377:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 3540.557572] Lustre: 230377:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 3540.560493] Lustre: 230377:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 3540.563573] Lustre: 230377:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 3540.568341] Lustre: 230377:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 3540.572598] Lustre: 230377:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 3540.575757] Lustre: 230377:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 3540.579926] Lustre: 230377:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 3540.583194] Lustre: 230377:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 3540.586468] Lustre: 230377:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 3549.664195] Lustre: 230656:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761325309/real 1761325309] req@000000006789b08c x1846883428335744/t0(0) o251->MGC192.168.201.127@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761325315 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3549.675861] Lustre: server umount MGS complete [ 3550.004517] loop0: detected capacity change from 0 to 204800000 [ 3550.040438] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3550.224399] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3550.243285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3550.259628] LustreError: 230710:0:(mdt_handler.c:7428:mdt_iocontrol()) lustre-MDT0000: Aborting recovery for device [ 3550.607355] loop1: detected capacity change from 0 to 204800000 [ 3550.649976] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3550.685718] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3550.722686] Lustre: 230825:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 3550.726845] Lustre: 230825:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 3550.729809] Lustre: 230825:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 3550.732803] Lustre: 230825:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 3550.736016] Lustre: 230825:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 3550.739578] Lustre: 230825:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 3550.742528] Lustre: 230825:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 3550.751114] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3550.825217] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3550.857188] LustreError: 230825:0:(mdt_handler.c:7428:mdt_iocontrol()) lustre-MDT0001: Aborting recovery for device [ 3551.191103] loop2: detected capacity change from 0 to 204800000 [ 3551.263675] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3551.298297] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3551.302683] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/loop2 [ 3551.321153] Lustre: 230911:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 3551.326621] Lustre: 230911:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 3551.333347] Lustre: 230911:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 3551.339358] Lustre: 230911:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 3551.346505] Lustre: 230911:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 3551.351561] Lustre: 230911:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 3551.356510] Lustre: 230911:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 3551.420779] Lustre: Found index 0 for lustre-OST0000, updating log [ 3551.459720] LustreError: 230911:0:(ofd_obd.c:1263:ofd_iocontrol()) lustre-OST0000: aborting recovery [ 3551.814533] loop3: detected capacity change from 0 to 204800000 [ 3551.858460] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3551.905071] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3551.910447] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/loop3 [ 3551.936203] Lustre: 231000:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 3551.941729] Lustre: 231000:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 3551.948835] Lustre: 231000:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 3551.953561] Lustre: 231000:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 3551.956839] Lustre: 231000:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 3551.960533] Lustre: 231000:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 3551.964570] Lustre: 231000:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 3552.001054] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3552.002996] Lustre: Skipped 1 previous similar message [ 3552.009914] LustreError: 231000:0:(ofd_obd.c:1263:ofd_iocontrol()) lustre-OST0001: aborting recovery [ 3555.043619] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6 to 0x0:33 [ 3555.052781] LustreError: 230768:0:(lod_dev.c:424:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000 get update log failed: rc = -108 [ 3555.053114] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6 to 0x0:33 [ 3556.066835] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:5 to 0x280000400:33 [ 3556.067582] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:5 to 0x2c0000400:33 [ 3557.355144] LustreError: 230733:0:(mdt_identity.c:112:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 346us: rc = -2 [ 3571.168754] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3571.171158] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3571.174706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3571.681366] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3571.682062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3571.685315] Lustre: Skipped 1 previous similar message [ 3574.725905] Lustre: server umount lustre-MDT0000 complete [ 3575.082603] LustreError: 230717:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761325341 with bad export cookie 11742666244724015384 [ 3575.084261] LustreError: 166-1: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3575.088821] LustreError: 230717:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3576.800924] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3576.801261] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3576.801698] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3576.801707] Lustre: Skipped 2 previous similar messages [ 3576.806099] LustreError: Skipped 1 previous similar message [ 3576.812291] Lustre: Skipped 2 previous similar messages [ 3580.832713] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3580.833287] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3580.836067] Lustre: Skipped 2 previous similar messages [ 3580.843410] LustreError: Skipped 1 previous similar message [ 3581.348287] Lustre: server umount lustre-MDT0001 complete [ 3587.772723] Lustre: server umount lustre-OST0000 complete [ 3602.400152] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3602.490969] Lustre: server umount lustre-OST0001 complete [ 3605.294321] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 13:02:50 (1761325370) [ 3614.898135] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [ 3618.005211] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 3621.560479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3623.764497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3625.822427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3627.993102] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3630.292909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3630.315935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3630.410712] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3630.421229] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3630.463483] Lustre: lustre-MDT0000: new disk, initializing [ 3630.489113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3630.496321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3631.965482] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 3636.369548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3636.393829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3636.429800] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3636.448967] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3636.452069] Lustre: Skipped 1 previous similar message [ 3636.495806] Lustre: lustre-MDT0001: new disk, initializing [ 3636.528497] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3636.540734] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3636.543430] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3638.004732] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 3641.940776] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3643.978126] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3646.303878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3646.329530] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3646.332143] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3646.425903] Lustre: lustre-OST0000: new disk, initializing [ 3646.427690] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3646.452226] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3648.046675] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 3650.038102] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3650.042343] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3652.078565] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3654.482058] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 3661.033590] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 3669.649519] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb in log lustre-client [ 3721.698062] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3721.698244] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3721.699648] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3721.704116] Lustre: Skipped 1 previous similar message [ 3721.707713] LustreError: Skipped 1 previous similar message [ 3722.737232] Lustre: server umount lustre-OST0000 complete [ 3726.817324] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3726.821580] Lustre: Skipped 1 previous similar message [ 3731.492171] LustreError: 235163:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761325497 with bad export cookie 11742666244724018058 [ 3731.495411] LustreError: 166-1: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3731.498598] LustreError: 235163:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3734.762819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3736.320869] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 3744.224177] Lustre: 240753:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761325504/real 1761325504] req@000000006bf56fb1 x1846883428400512/t0(0) o251->MGC192.168.201.127@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761325510 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 3744.238524] Lustre: server umount MGS complete [ 3744.240066] Lustre: Skipped 2 previous similar messages [ 3748.093038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3748.263854] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3748.271796] LustreError: 241386:0:(qmt_pool.c:1415:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 3748.291173] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3749.779399] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 3752.751839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3754.410074] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 3756.941904] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3758.925852] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3761.292457] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3762.949769] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 3765.569313] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3780.064777] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3780.067928] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3780.072891] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3780.075023] Lustre: Skipped 3 previous similar messages [ 3784.162491] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3785.833748] Lustre: server umount lustre-OST0000 complete [ 3788.768723] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3794.402182] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3794.407363] LustreError: Skipped 1 previous similar message [ 3794.505270] LustreError: 241356:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761325560 with bad export cookie 11742666244724019955 [ 3794.508382] LustreError: 166-1: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3794.508896] LustreError: 241356:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3799.459044] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 3800.537172] Key type lgssc unregistered [ 3800.660368] LNet: 245485:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3800.663162] LNet: Removed LNI 192.168.201.127@tcp [ 3800.967668] Key type .llcrypt unregistered [ 3800.969833] Key type ._llcrypt unregistered [ 3814.270425] alg: No test for adler32 (adler32-zlib) [ 3815.022343] Key type ._llcrypt registered [ 3815.024100] Key type .llcrypt registered [ 3815.063840] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [ 3820.182807] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 3820.609595] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3820.936387] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 3821.031222] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 3821.032821] LNet: Accept secure, port 988 [ 3822.616176] Key type lgssc registered [ 3823.029463] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3826.125788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3828.266283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3830.223700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3832.296951] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3834.446764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3834.473073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3835.584797] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3835.596477] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3835.633477] Lustre: lustre-MDT0000: new disk, initializing [ 3835.656658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3835.663242] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3837.169050] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 3841.401129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3841.424532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3841.457035] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3841.474318] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3841.477640] Lustre: Skipped 1 previous similar message [ 3841.497690] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3841.521820] Lustre: lustre-MDT0001: new disk, initializing [ 3841.553190] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3841.566086] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3841.568966] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3843.048227] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 3847.020680] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3849.047562] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3851.439351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3851.464445] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3851.466844] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3851.583963] Lustre: lustre-OST0000: new disk, initializing [ 3851.585866] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3851.614466] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3853.155440] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 3857.233233] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3859.449150] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3859.453442] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3866.147152] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 3870.005854] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 3874.608873] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb in log lustre-client [ 3920.865952] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3920.866333] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3920.867976] LustreError: Skipped 1 previous similar message [ 3920.871338] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3920.873648] Lustre: Skipped 1 previous similar message [ 3924.645209] Lustre: server umount lustre-OST0000 complete [ 3925.984871] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3925.985363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3925.988683] Lustre: Skipped 2 previous similar messages [ 3931.104859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3931.107875] Lustre: Skipped 2 previous similar messages [ 3936.225215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3936.227697] Lustre: Skipped 1 previous similar message [ 3940.320112] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3940.361614] Lustre: server umount lustre-MDT0000 complete [ 3941.517979] LustreError: 249568:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761325707 with bad export cookie 16190764672949983827 [ 3941.520245] LustreError: 166-1: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3941.523111] LustreError: 249568:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3941.635972] Lustre: server umount lustre-MDT0001 complete [ 3944.379560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3945.821844] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 3953.632123] Lustre: 254921:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761325713/real 1761325713] req@00000000f5c169fb x1846883738759360/t0(0) o251->MGC192.168.201.127@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761325719 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3953.642703] Lustre: server umount MGS complete [ 3957.474892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3957.656118] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3957.666212] LustreError: 255555:0:(qmt_pool.c:1415:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 3957.691764] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3959.164816] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 3961.848396] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3961.949683] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3963.276778] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 3965.729987] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3967.588101] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3969.866955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3969.979118] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3971.471628] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 3974.104328] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3992.546108] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3992.547376] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3992.550291] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3992.555860] Lustre: Skipped 1 previous similar message [ 3992.559155] Lustre: Skipped 2 previous similar messages [ 3995.559690] Lustre: server umount lustre-OST0000 complete [ 3997.666228] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3997.666411] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4004.397986] LustreError: 255525:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761325770 with bad export cookie 16190764672949985724 [ 4004.400345] LustreError: 166-1: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4004.401503] LustreError: 255525:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4004.516978] Lustre: server umount lustre-MDT0001 complete [ 4004.518301] Lustre: Skipped 1 previous similar message [ 4009.343106] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 4010.425397] Key type lgssc unregistered [ 4010.540404] LNet: 259654:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4010.543391] LNet: Removed LNI 192.168.201.127@tcp [ 4010.858571] Key type .llcrypt unregistered [ 4010.860386] Key type ._llcrypt unregistered [ 4021.360227] alg: No test for adler32 (adler32-zlib) [ 4022.112369] Key type ._llcrypt registered [ 4022.113594] Key type .llcrypt registered [ 4022.151219] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 4022.607097] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4022.780749] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 4022.865203] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 4022.867515] LNet: Accept secure, port 988 [ 4024.456104] Key type lgssc registered [ 4024.842596] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4027.578084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4028.720446] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4028.735833] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4030.060936] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 4032.737172] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4032.810614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4032.927317] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4034.305082] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 4036.881937] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4038.829652] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4041.153793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4041.252729] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4042.637375] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 4045.216545] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4047.471070] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4047.491871] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4047.494352] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 4047.540653] Lustre: lustre-OST0001: new disk, initializing [ 4047.542185] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 4047.566586] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4048.925656] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 4053.762194] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 4053.851087] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4054.527395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4054.531633] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4054.551087] LustreError: 264986:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 4055.588101] LustreError: 264986:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 4055.592817] LustreError: 264986:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [ 4055.851282] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9e24917f3000.ost_server_uuid 40 [ 4056.345080] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9e24917f3000.ost_server_uuid in FULL state after 0 sec [ 4056.611570] LustreError: 264986:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 4056.614925] LustreError: 264986:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [ 4058.149591] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9e24917f3000.ost_server_uuid 40 [ 4058.632979] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9e24917f3000.ost_server_uuid in FULL state after 0 sec [ 4058.658129] LustreError: 264986:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 4058.662153] LustreError: 264986:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) Skipped 3 previous similar messages [ 4072.743802] Lustre: lustre-OST0001: Not available for connect from 192.168.201.27@tcp (stopping) [ 4074.980194] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4074.980273] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4074.982104] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4074.984727] LustreError: Skipped 1 previous similar message [ 4074.997981] Lustre: Skipped 1 previous similar message [ 4076.410248] Lustre: server umount lustre-OST0001 complete [ 4079.073322] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4079.075707] LustreError: Skipped 1 previous similar message [ 4079.077035] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4079.081235] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4079.083055] Lustre: Skipped 1 previous similar message [ 4080.098900] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4080.102749] LustreError: Skipped 1 previous similar message [ 4084.262678] Lustre: server umount lustre-OST0000 complete [ 4089.312715] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4089.315462] LustreError: Skipped 1 previous similar message [ 4089.316686] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4089.319744] Lustre: Skipped 1 previous similar message [ 4089.321571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4089.323331] Lustre: Skipped 3 previous similar messages [ 4091.718319] Lustre: server umount lustre-MDT0000 complete [ 4093.123261] LustreError: 261198:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761325859 with bad export cookie 14284301001781158730 [ 4093.125834] LustreError: 166-1: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4093.128655] LustreError: 261198:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4097.980527] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 4098.985573] Key type lgssc unregistered [ 4099.097702] LNet: 266868:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4099.100828] LNet: Removed LNI 192.168.201.127@tcp [ 4099.443529] Key type .llcrypt unregistered [ 4099.444740] Key type ._llcrypt unregistered [ 4113.232358] alg: No test for adler32 (adler32-zlib) [ 4113.984370] Key type ._llcrypt registered [ 4113.985775] Key type .llcrypt registered [ 4114.027823] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 4114.521493] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4114.666036] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 4114.739571] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 4114.741713] LNet: Accept secure, port 988 [ 4116.328090] Key type lgssc registered [ 4116.649110] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4119.553346] loop0: detected capacity change from 0 to 1099511627776 [ 4119.702221] SGI XFS with ACLs, security attributes, quota, no debug enabled [ 4119.706085] *** VALIDATE xfs *** [ 4119.712506] XFS (loop0): Mounting V5 Filesystem [ 4119.726701] XFS (loop0): Ending clean mount [ 4120.181745] loop1: detected capacity change from 0 to 18691697672192 [ 4136.042900] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4141.557561] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [ 4144.596331] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 4147.803357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4149.861486] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4151.825027] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4154.032396] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4156.350081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4156.377905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4157.469765] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4157.481163] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4157.512449] Lustre: lustre-MDT0000: new disk, initializing [ 4157.540800] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4157.550850] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4158.936648] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 4163.119399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4163.141226] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4163.178857] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4163.210654] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4163.212686] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 4163.216781] Lustre: Skipped 1 previous similar message [ 4163.255933] Lustre: lustre-MDT0001: new disk, initializing [ 4163.287515] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4163.301155] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4163.304660] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4164.876459] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 4168.693533] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4170.573327] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4172.767113] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4172.793907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4172.796339] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4172.890552] Lustre: lustre-OST0000: new disk, initializing [ 4172.892504] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 4172.923920] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4174.441718] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 4178.223593] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4178.421800] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4178.426547] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4179.681313] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 4179.758285] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4181.119938] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 4181.200901] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4183.522475] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4183.522607] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4183.525648] LustreError: Skipped 1 previous similar message [ 4183.526248] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4183.532511] Lustre: Skipped 1 previous similar message [ 4187.943410] Lustre: server umount lustre-OST0000 complete [ 4193.761504] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4193.762142] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4193.766301] Lustre: Skipped 1 previous similar message [ 4193.768084] Lustre: Skipped 1 previous similar message [ 4195.457420] Lustre: server umount lustre-MDT0000 complete [ 4196.738252] LustreError: 271797:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761325963 with bad export cookie 15107706014246482644 [ 4196.740545] LustreError: 166-1: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4196.743792] LustreError: 271797:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4196.862056] Lustre: server umount lustre-MDT0001 complete [ 4199.808564] XFS (loop0): Unmounting Filesystem [ 4201.451273] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 13:12:46 (1761325966) [ 4203.771347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4203.922838] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4203.944321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4205.412854] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 4208.311645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4208.428327] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4209.961770] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 4212.484410] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4214.434682] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4216.796070] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4216.907082] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4218.280119] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 4220.709592] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4229.600910] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4229.604408] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4229.609163] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4233.698319] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4233.698667] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4233.701918] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4234.935130] Lustre: server umount lustre-OST0000 complete [ 4238.817768] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4242.503549] Lustre: server umount lustre-MDT0000 complete [ 4243.744308] LustreError: 276481:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761326010 with bad export cookie 15107706014246483498 [ 4243.746338] LustreError: 166-1: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4243.749819] LustreError: 276481:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4248.440625] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 4249.476544] Key type lgssc unregistered [ 4249.597416] LNet: 280249:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4249.600075] LNet: Removed LNI 192.168.201.127@tcp [ 4249.909342] Key type .llcrypt unregistered [ 4249.910468] Key type ._llcrypt unregistered [ 4259.763901] alg: No test for adler32 (adler32-zlib) [ 4260.518682] Key type ._llcrypt registered [ 4260.519684] Key type .llcrypt registered [ 4260.563051] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 4261.087948] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4261.286309] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 4261.370576] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 4261.372492] LNet: Accept secure, port 988 [ 4262.960186] Key type lgssc registered [ 4263.342419] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4265.971174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4267.112425] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4267.130213] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4268.560917] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 4271.272596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4271.388487] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4272.792273] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 4275.063100] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4276.835459] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4278.905374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4278.999323] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4280.443218] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 4283.173251] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4291.553612] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4291.560418] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4291.562180] Lustre: Skipped 1 previous similar message [ 4294.953360] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.27@tcp (stopping) [ 4296.673280] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4296.676545] Lustre: Skipped 1 previous similar message [ 4296.911344] Lustre: server umount lustre-MDT0001 complete [ 4298.018881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4300.065330] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 192.168.201.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4300.839966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4300.879816] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4300.883081] Lustre: Found index 1 for lustre-MDT0001, updating log [ 4300.900685] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4301.014164] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4302.513338] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 4306.404354] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 4306.408060] LustreError: 285664:0:(genops.c:522:class_register_device()) lustre-MDT0001-lwp-OST0000: already exists, won't add [ 4306.411870] LustreError: 285664:0:(obd_mount.c:195:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 4306.413961] LustreError: 285664:0:(obd_mount_server.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 4306.416211] LustreError: 281855:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 4306.417729] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 4306.421170] Lustre: Skipped 1 previous similar message [ 4306.432257] LustreError: 285668:0:(obd_config.c:1999:class_config_llog_handler()) MGC192.168.201.127@tcp: cfg command failed: rc = -17 [ 4306.436363] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 4306.436363] [ 4607.271284] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.27@tcp (stopping) [ 4608.481416] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 4608.482405] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4608.488586] Lustre: Skipped 2 previous similar messages [ 4612.385172] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.27@tcp (stopping) [ 4612.388906] Lustre: Skipped 2 previous similar messages [ 4619.232139] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 4619.274279] Lustre: server umount lustre-MDT0001 complete [ 4620.225245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4622.625510] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 192.168.201.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4622.704873] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4622.736390] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4622.738046] Lustre: Found index 1 for lustre-MDT0001, updating log [ 4622.751617] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4622.862976] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4624.314216] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 4627.939116] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 4627.945646] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 4627.947473] Lustre: Skipped 1 previous similar message [ 4632.549571] LustreError: 298446:0:(genops.c:522:class_register_device()) lustre-MDT0001-lwp-OST0000: already exists, won't add [ 4632.553012] LustreError: 298446:0:(genops.c:522:class_register_device()) Skipped 1 previous similar message [ 4632.555946] LustreError: 298446:0:(obd_mount.c:195:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 4632.559592] LustreError: 298446:0:(obd_mount_server.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 4632.563423] LustreError: 281855:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 4632.565259] LustreError: 281855:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [ 4632.579340] LustreError: 298449:0:(obd_config.c:680:class_setup()) Device 8 already setup (type osp) [ 4632.581499] LustreError: 298449:0:(obd_config.c:1999:class_config_llog_handler()) MGC192.168.201.127@tcp: cfg command failed: rc = -17 [ 4632.584038] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.201.127@tcp [ 4632.584038] [ 4929.831853] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.27@tcp (stopping) [ 4929.834534] Lustre: Skipped 5 previous similar messages [ 4930.017955] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 4930.018508] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4930.025179] Lustre: Skipped 1 previous similar message [ 4941.280113] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 4941.326459] Lustre: server umount lustre-MDT0001 complete [ 4942.222690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4944.733162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4944.764538] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4944.766238] Lustre: Found index 1 for lustre-MDT0001, updating log [ 4944.779265] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4944.874720] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 4946.257660] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 4949.989946] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 4949.999848] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to (at 0@lo) [ 4950.001678] Lustre: Skipped 1 previous similar message [ 4953.062372] LustreError: 311080:0:(genops.c:522:class_register_device()) lustre-MDT0001-lwp-OST0000: already exists, won't add [ 4953.066239] LustreError: 311080:0:(obd_mount.c:195:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 4953.069778] LustreError: 311080:0:(obd_mount_server.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 4953.073590] LustreError: 281855:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 4953.075224] LustreError: 281855:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [ 4953.089101] LustreError: 311083:0:(lod_lov.c:228:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [ 4953.093886] LustreError: 311083:0:(obd_config.c:1999:class_config_llog_handler()) MGC192.168.201.127@tcp: cfg command failed: rc = -114 [ 4953.096134] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [ 4953.096134] [ 5262.308122] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5262.308428] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5262.309511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5262.309520] Lustre: Skipped 8 previous similar messages [ 5262.316413] Lustre: Skipped 2 previous similar messages [ 5266.501727] Lustre: server umount lustre-MDT0000 complete [ 5267.426675] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5267.430652] LustreError: Skipped 2 previous similar messages [ 5267.752212] LustreError: 297713:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761327034 with bad export cookie 1153121448254530282 [ 5267.755022] LustreError: 166-1: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5267.755876] LustreError: 297713:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5272.544819] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5272.544992] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5274.593207] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5279.712946] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5282.272225] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 5282.389181] Lustre: server umount lustre-MDT0001 complete [ 5286.678096] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [ 5289.525332] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 5292.851652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5295.005389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5297.022419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5299.021066] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5301.137974] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 13:31:06 (1761327066) [ 5303.376686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5303.399701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5303.478346] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5303.488874] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5303.524765] Lustre: lustre-MDT0000: new disk, initializing [ 5303.547177] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5303.553572] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5305.018160] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 5309.272228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5309.297958] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5309.335944] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5309.354273] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 5309.356365] Lustre: Skipped 1 previous similar message [ 5309.391072] Lustre: lustre-MDT0001: new disk, initializing [ 5309.419548] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5309.430453] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5309.434117] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5310.767191] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 5314.479267] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5316.275967] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5318.551531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5318.582269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5318.585935] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 5318.691812] Lustre: lustre-OST0000: new disk, initializing [ 5318.694507] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5318.720904] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5320.222423] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all 8 [ 5321.079713] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5321.082518] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5324.000492] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5329.889198] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5329.893528] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5329.897346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5329.899894] Lustre: Skipped 5 previous similar messages [ 5330.985978] Lustre: server umount lustre-MDT0000 complete [ 5330.988462] Lustre: Skipped 1 previous similar message [ 5331.425708] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5331.431365] LustreError: Skipped 2 previous similar messages [ 5332.257151] LustreError: 327767:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761327098 with bad export cookie 1153121448254532179 [ 5332.260180] LustreError: 326547:0:(osp_object.c:629:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x2:0x0]: rc = -5 [ 5332.261293] LustreError: 166-1: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5332.264710] LustreError: 326547:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) lustre-MDT0000-osp-MDT0001: fail to cancel 1 llog-records: rc = -5 [ 5332.271516] LustreError: 326547:0:(llog_cat.c:789:llog_cat_cancel_records()) lustre-MDT0000-osp-MDT0001: fail to cancel 1 of 1 llog-records: rc = -5 [ 5339.104152] Lustre: 328935:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761327099/real 1761327099] req@000000002f013398 x1846884200264064/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761327105 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 5348.585145] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [ 5351.404954] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 5355.022384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5357.688267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5359.829227] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5362.089311] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5362.623549] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 5212 sec ========== 13:32:07 (1761327127) [ 5377.212679] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 5378.340289] Key type lgssc unregistered [ 5378.455306] LNet: 334030:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5378.458372] LNet: Removed LNI 192.168.201.127@tcp [ 5378.788195] Key type .llcrypt unregistered [ 5378.789246] Key type ._llcrypt unregistered