[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 609548933 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001016] APIC: Switch to symmetric I/O mode setup [ 0.003232] x2apic enabled [ 0.004006] Switched APIC routing to physical x2apic. [ 0.005018] kvm-guest: setup PV IPIs [ 0.006000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.006000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.006000] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.006022] pid_max: default: 32768 minimum: 301 [ 0.007195] LSM: Security Framework initializing [ 0.008102] Yama: becoming mindful. [ 0.009037] SELinux: Initializing. [ 0.010091] *** VALIDATE selinux *** [ 0.015444] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.022488] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.024090] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025150] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026108] *** VALIDATE tmpfs *** [ 0.027525] *** VALIDATE proc *** [ 0.028215] *** VALIDATE cgroup *** [ 0.029006] *** VALIDATE cgroup2 *** [ 0.030335] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.031162] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.032008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.033044] Spectre V2 : User space: Vulnerable [ 0.034013] Speculative Store Bypass: Vulnerable [ 0.037419] debug: unmapping init [mem 0xffffffffa7859000-0xffffffffa7860fff] [ 0.039882] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.040867] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.041022] ... version: 2 [ 0.042006] ... bit width: 48 [ 0.043007] ... generic registers: 4 [ 0.044007] ... value mask: 0000ffffffffffff [ 0.045006] ... max period: 00007fffffffffff [ 0.046007] ... fixed-purpose events: 3 [ 0.047006] ... event mask: 000000070000000f [ 0.048258] rcu: Hierarchical SRCU implementation. [ 0.050440] smp: Bringing up secondary CPUs ... [ 0.051485] x86: Booting SMP configuration: [ 0.052014] .... node #0, CPUs: #1 #2 #3 [ 0.060052] smp: Brought up 1 node, 4 CPUs [ 0.062016] smpboot: Max logical packages: 1 [ 0.063025] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.107584] node 0 deferred pages initialised in 41ms [ 0.111284] devtmpfs: initialized [ 0.112267] x86/mm: Memory block size: 128MB [ 0.120034] gcov: version magic: 0x41383552 [ 0.123529] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.126189] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.129545] pinctrl core: initialized pinctrl subsystem [ 0.132513] [ 0.133007] ************************************************************* [ 0.137013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.140008] ** ** [ 0.146021] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.149009] ** ** [ 0.151007] ** This means that this kernel is built to expose internal ** [ 0.153009] ** IOMMU data structures, which may compromise security on ** [ 0.156050] ** your system. ** [ 0.161048] ** ** [ 0.164011] ** If you see this message and you are not debugging the ** [ 0.167020] ** kernel, report this immediately to your vendor! ** [ 0.168025] ** ** [ 0.171012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.173010] ************************************************************* [ 0.177677] NET: Registered protocol family 16 [ 0.179403] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.184061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.194082] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.199233] cpuidle: using governor menu [ 0.203545] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.212477] PCI: Using configuration type 1 for base access [ 0.214316] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.248116] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.249042] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.251000] cryptd: max_cpu_qlen set to 1000 [ 0.258386] ACPI: Added _OSI(Module Device) [ 0.259026] ACPI: Added _OSI(Processor Device) [ 0.265043] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.268025] ACPI: Added _OSI(Processor Aggregator Device) [ 0.278620] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.297698] ACPI: Interpreter enabled [ 0.301135] ACPI: PM: (supports S0 S3 S4 S5) [ 0.303010] ACPI: Using IOAPIC for interrupt routing [ 0.306145] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.312000] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.325598] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.327033] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.328015] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.331123] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.338206] acpiphp: Slot [2] registered [ 0.341119] acpiphp: Slot [3] registered [ 0.342189] acpiphp: Slot [4] registered [ 0.345422] acpiphp: Slot [5] registered [ 0.347117] acpiphp: Slot [6] registered [ 0.350029] acpiphp: Slot [7] registered [ 0.353640] acpiphp: Slot [8] registered [ 0.355193] acpiphp: Slot [9] registered [ 0.357086] acpiphp: Slot [10] registered [ 0.359082] acpiphp: Slot [11] registered [ 0.365136] acpiphp: Slot [12] registered [ 0.368092] acpiphp: Slot [13] registered [ 0.369082] acpiphp: Slot [14] registered [ 0.373000] acpiphp: Slot [15] registered [ 0.373000] acpiphp: Slot [16] registered [ 0.378022] acpiphp: Slot [17] registered [ 0.380096] acpiphp: Slot [18] registered [ 0.382124] acpiphp: Slot [19] registered [ 0.383000] acpiphp: Slot [20] registered [ 0.386145] acpiphp: Slot [21] registered [ 0.389171] acpiphp: Slot [22] registered [ 0.391274] acpiphp: Slot [23] registered [ 0.392123] acpiphp: Slot [24] registered [ 0.396094] acpiphp: Slot [25] registered [ 0.398086] acpiphp: Slot [26] registered [ 0.407036] acpiphp: Slot [27] registered [ 0.411139] acpiphp: Slot [28] registered [ 0.417160] acpiphp: Slot [29] registered [ 0.422138] acpiphp: Slot [30] registered [ 0.432234] acpiphp: Slot [31] registered [ 0.434128] PCI host bridge to bus 0000:00 [ 0.436024] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.438031] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.442022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.443019] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.444014] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.445018] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.446178] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.452399] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.454505] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.464000] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.470053] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.472029] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.482045] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.491027] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.510121] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.511825] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.513060] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.514000] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.518026] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.527025] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.537031] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.541000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.545000] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.548026] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.579026] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.596000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.615022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.633024] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.662021] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.689562] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.706024] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.723025] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.772023] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.791090] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.806024] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.810023] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.830021] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.867703] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.877018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.884016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.907020] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.922000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.931026] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.940021] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.974022] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 1.001000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 1.002516] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 1.005363] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 1.008540] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 1.011225] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 1.015120] iommu: Default domain type: Passthrough [ 1.016000] SCSI subsystem initialized [ 1.016000] ACPI: bus type USB registered [ 1.025161] usbcore: registered new interface driver usbfs [ 1.031226] usbcore: registered new interface driver hub [ 1.037107] usbcore: registered new device driver usb [ 1.041339] pps_core: LinuxPPS API ver. 1 registered [ 1.045013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.050122] PTP clock support registered [ 1.054105] EDAC MC: Ver: 3.0.0 [ 1.060147] PCI: Using ACPI for IRQ routing [ 1.067360] NetLabel: Initializing [ 1.070012] NetLabel: domain hash size = 128 [ 1.078018] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.084125] NetLabel: unlabeled traffic allowed by default [ 1.095108] vgaarb: loaded [ 1.100341] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.101000] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.118093] clocksource: Switched to clocksource kvm-clock [ 1.433250] VFS: Disk quotas dquot_6.6.0 [ 1.436312] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.441915] *** VALIDATE ramfs *** [ 1.442800] *** VALIDATE hugetlbfs *** [ 1.451537] pnp: PnP ACPI init [ 1.455913] pnp: PnP ACPI: found 6 devices [ 1.483284] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.485617] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.487015] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.492672] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.494236] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.495742] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 1.502972] NET: Registered protocol family 2 [ 1.504678] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.520705] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.531276] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.546686] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.557796] TCP: Hash tables configured (established 65536 bind 65536) [ 1.564719] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.571592] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.577034] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.586795] NET: Registered protocol family 1 [ 1.593463] RPC: Registered named UNIX socket transport module. [ 1.598241] RPC: Registered udp transport module. [ 1.604288] RPC: Registered tcp transport module. [ 1.605867] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.614476] NET: Registered protocol family 44 [ 1.615562] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.621022] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.624352] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.627779] PCI: CLS 0 bytes, default 64 [ 1.631819] Unpacking initramfs... [ 5.658815] debug: unmapping init [mem 0xffff8d44fcc54000-0xffff8d44fffbffff] [ 5.669419] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 5.674175] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 5.681979] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 7.381030] Initialise system trusted keyrings [ 7.382773] Key type blacklist registered [ 7.404840] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 7.438203] zbud: loaded [ 7.449199] *** VALIDATE nfs *** [ 7.458663] *** VALIDATE nfs4 *** [ 7.467482] pstore: using deflate compression [ 7.492923] Platform Keyring initialized [ 7.832888] NET: Registered protocol family 38 [ 7.834095] Key type asymmetric registered [ 7.835114] Asymmetric key parser 'x509' registered [ 7.836738] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 7.855358] io scheduler mq-deadline registered [ 7.856970] io scheduler kyber registered [ 7.858586] io scheduler bfq registered [ 7.868686] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 7.871364] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 7.873920] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 7.880948] ACPI: Power Button [PWRF] [ 8.371277] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 8.841833] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 9.716360] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 10.107357] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 11.304646] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 11.457021] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 11.600174] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 11.623518] Non-volatile memory driver v1.3 [ 11.624826] Linux agpgart interface v0.103 [ 12.003593] virtio_blk virtio1: [vda] 68256 512-byte logical blocks (34.9 MB/33.3 MiB) [ 12.030952] vda: detected capacity change from 0 to 34947072 [ 12.111161] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 12.115249] vdb: detected capacity change from 0 to 1073741824 [ 12.141299] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 12.147853] vdc: detected capacity change from 0 to 2621440000 [ 12.198558] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 12.201421] vdd: detected capacity change from 0 to 2621440000 [ 12.250086] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 12.258522] vde: detected capacity change from 0 to 4294967296 [ 12.317449] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 12.323525] vdf: detected capacity change from 0 to 4294967296 [ 12.334788] libphy: Fixed MDIO Bus: probed [ 12.368173] usbcore: registered new interface driver usbserial_generic [ 12.377486] usbserial: USB Serial support registered for generic [ 12.383382] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 12.395453] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 12.404981] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 12.415043] mousedev: PS/2 mouse device common for all mice [ 12.428102] rtc_cmos 00:05: RTC can wake from S4 [ 12.434131] rtc_cmos 00:05: registered as rtc0 [ 12.436108] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 12.438649] intel_pstate: CPU model not supported [ 12.454456] hid: raw HID events driver (C) Jiri Kosina [ 12.456556] usbcore: registered new interface driver usbhid [ 12.458510] usbhid: USB HID core driver [ 12.460059] drop_monitor: Initializing network drop monitor service [ 12.462391] Initializing XFRM netlink socket [ 12.464346] NET: Registered protocol family 10 [ 12.466640] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 12.470677] Segment Routing with IPv6 [ 12.472072] NET: Registered protocol family 17 [ 12.479816] mpls_gso: MPLS GSO support [ 12.485504] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 12.541585] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 12.545361] RAS: Correctable Errors collector initialized. [ 12.546788] AVX version of gcm_enc/dec engaged. [ 12.547938] AES CTR mode by8 optimization enabled [ 12.703313] sched_clock: Marking stable (12703294711, 0)->(14851509387, -2148214676) [ 12.707633] registered taskstats version 1 [ 12.709769] Loading compiled-in X.509 certificates [ 12.711885] zswap: loaded using pool lzo/zbud [ 12.869235] Key type big_key registered [ 12.894550] Key type encrypted registered [ 12.896533] ima: No TPM chip found, activating TPM-bypass! [ 12.900067] ima: Allocated hash algorithm: sha1 [ 12.901558] ima: No architecture policies found [ 12.905270] evm: Initialising EVM extended attributes: [ 12.906843] evm: security.selinux [ 12.907891] evm: security.ima [ 12.908831] evm: security.capability [ 12.910127] evm: HMAC attrs: 0x1 [ 12.913459] rtc_cmos 00:05: setting system clock to 2025-10-14 14:22:39 UTC (1760451759) [ 12.929432] debug: unmapping init [mem 0xffffffffa8803000-0xffffffffa89fffff] [ 12.932548] debug: unmapping init [mem 0xffffffffa7582000-0xffffffffa7858fff] [ 12.942086] Write protecting the kernel read-only data: 28672k [ 12.949966] debug: unmapping init [mem 0xffffffffa5c03000-0xffffffffa5dfffff] [ 12.955221] debug: unmapping init [mem 0xffffffffa6514000-0xffffffffa65fffff] [ 13.036891] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 13.083646] systemd[1]: Detected virtualization kvm. [ 13.085882] systemd[1]: Detected architecture x86-64. [ 13.087670] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 13.125922] systemd[1]: No hostname configured. [ 13.127575] systemd[1]: Set hostname to . [ 13.129304] random: systemd: uninitialized urandom read (16 bytes read) [ 13.133264] systemd[1]: Initializing machine ID from random generator. [ 13.406429] random: systemd: uninitialized urandom read (16 bytes read) [ 13.408844] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 13.438783] random: systemd: uninitialized urandom read (16 bytes read) [ 13.441474] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 13.446194] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Swap. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 15.162949] device-mapper: uevent: version 1.0.3 [ 15.165269] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 17.327978] virtio_net virtio0 ens2: renamed from eth0 [ 17.886258] random: fast init done [ 18.433767] scsi host0: ata_piix [ 18.687478] scsi host1: ata_piix [ 18.762581] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 18.764615] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 23.428309] random: crng init done [ 23.429757] random: 7 urandom warning(s) missed due to ratelimiting [ 26.381860] dracut-initqueue[580]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ 28.620953] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Reached target Remote File Systems. [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 31.700677] printk: systemd: 26 output lines suppressed due to ratelimiting [ 32.495185] SELinux: Disabled at runtime. [ 32.608387] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 32.632211] systemd[1]: Detected virtualization kvm. [ 32.645823] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 33.891265] systemd[1]: initrd-switch-root.service: Succeeded. [ 33.897697] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 33.908820] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 33.928580] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 33.932013] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 33.946443] systemd[1]: Starting Journal Service... Starting Journal Service... [ 33.959477] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Switch Root. Mounting Huge Pages File System... [ OK ] Stopped target Initrd File Systems. Mounting POSIX Message Queue File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-getty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting Kernel Debug File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. Activating swap /dev/disk/by-label/SWAP... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ 34.441562] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice User and Session Slice. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. Starting Remount Root and Kernel File Systems... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 35.242243] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ 35.389279] hrtimer: interrupt took 2481258 ns [ OK ] Started udev Kernel Device Manager. [ 36.204609] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 36.291296] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 36.664781] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 36.796196] EDAC sbridge: Ver: 1.1.2 [ 39.937595] Key type dns_resolver registered [ 40.954393] NFS: Registering the id_resolver key type [ 40.956215] Key type id_resolver registered [ 40.979113] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (7s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg349-server login: [ 68.828408] libcfs: loading out-of-tree module taints kernel. [ 68.846695] alg: No test for adler32 (adler32-zlib) [ 69.599271] Key type ._llcrypt registered [ 69.600903] Key type .llcrypt registered [ 69.640589] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_hostid [ 77.229889] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 77.882441] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 78.208927] Lustre: Lustre: Build Version: 2.15.7_5_g4a1d30d [ 78.538518] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 78.541887] LNet: Accept secure, port 988 [ 80.184117] Key type lgssc registered [ 80.719450] Lustre: Echo OBD driver; http://www.lustre.org/ [ 87.044414] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 88.235706] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 92.549262] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 95.803893] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 99.207416] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 106.006654] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 111.735743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 111.794049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 112.914195] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 112.929214] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 112.975316] Lustre: lustre-MDT0000: new disk, initializing [ 113.029637] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 113.038949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 114.811707] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 120.837672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 120.878075] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 120.913493] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 120.926285] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 120.928441] Lustre: Skipped 1 previous similar message [ 120.971655] Lustre: lustre-MDT0001: new disk, initializing [ 121.000893] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 121.014669] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 121.020574] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 122.739865] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 128.483459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 128.516744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 128.520150] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 128.614522] Lustre: lustre-OST0000: new disk, initializing [ 128.616560] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 128.634648] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 130.164199] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 131.057601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 131.062082] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 135.583506] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 135.619846] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 135.624298] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 135.667459] Lustre: lustre-OST0001: new disk, initializing [ 135.670040] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 135.690261] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 136.310653] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 136.315499] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 137.381578] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 144.190565] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 149.332075] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 155.767883] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing check_logdir /tmp/testlogs/ [ 157.529191] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing yml_node [ 159.467300] Lustre: DEBUG MARKER: Client: 2.15.7.5 [ 160.726612] Lustre: DEBUG MARKER: MDS: 2.15.7.5 [ 162.188950] Lustre: DEBUG MARKER: OSS: 2.15.7.5 [ 163.130375] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Oct 14 10:25:08 EDT 2025 [ 167.493665] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball 110 [ 168.220847] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 [ 182.752946] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 182.753223] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 182.753297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 182.758511] Lustre: Skipped 2 previous similar messages [ 186.466237] Lustre: server umount lustre-MDT0000 complete [ 187.872797] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 187.878973] LustreError: Skipped 3 previous similar messages [ 188.434370] LustreError: 6213:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760451935 with bad export cookie 5601075482732419037 [ 188.434705] LustreError: 166-1: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 188.440669] LustreError: 6213:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 192.992915] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 192.997307] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 192.997514] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 192.997519] Lustre: Skipped 3 previous similar messages [ 193.009591] Lustre: Skipped 2 previous similar messages [ 193.024813] LustreError: Skipped 1 previous similar message [ 194.019046] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 194.019061] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 194.023967] Lustre: Skipped 2 previous similar messages [ 194.029626] LustreError: Skipped 1 previous similar message [ 194.640291] Lustre: server umount lustre-MDT0001 complete [ 196.625643] Lustre: server umount lustre-OST0000 complete [ 201.682851] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_hostid [ 205.730474] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 210.475095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 213.930457] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 217.225301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 220.669496] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 224.108184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 224.152958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 224.295882] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 224.319154] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 224.364483] Lustre: lustre-MDT0000: new disk, initializing [ 224.413134] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 224.422952] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 226.451950] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 232.315579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 232.350814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 232.394207] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 232.419793] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 232.422585] Lustre: Skipped 1 previous similar message [ 232.466413] Lustre: lustre-MDT0001: new disk, initializing [ 232.512789] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 232.536275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 232.542113] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 234.692141] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 239.913576] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 242.874536] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 246.478300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 246.519576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 246.524191] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 247.958235] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 248.849957] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 254.317873] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 256.694639] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 256.835658] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 259.034882] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 259.159315] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 263.648875] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 263.650031] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 263.653344] LustreError: Skipped 1 previous similar message [ 263.660418] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 263.664347] Lustre: Skipped 1 previous similar message [ 266.439776] Lustre: server umount lustre-OST0000 complete [ 266.443436] Lustre: Skipped 1 previous similar message [ 268.772901] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 268.779873] Lustre: Skipped 1 previous similar message [ 268.783225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 273.377098] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 273.379923] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 276.583624] Lustre: server umount lustre-MDT0000 complete [ 278.476846] LustreError: 15155:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760452025 with bad export cookie 5601075482732420493 [ 278.480178] LustreError: 166-1: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 278.483779] LustreError: 15155:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 280.967368] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 10:27:06 (1760452026) [ 284.406285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 284.654339] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 284.695085] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 284.699043] Lustre: Skipped 1 previous similar message [ 286.759460] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 289.762345] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 290.844073] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 293.217321] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 297.004894] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 299.650602] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 302.897706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 305.214256] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 308.748285] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 321.610610] Lustre: Modifying parameter general.osc.*.max_dirty_mb in log params [ 325.604219] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 325.605421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 325.611820] Lustre: Skipped 2 previous similar messages [ 325.614078] Lustre: Skipped 3 previous similar messages [ 331.353058] Lustre: server umount lustre-MDT0000 complete [ 331.354925] Lustre: Skipped 1 previous similar message [ 332.972406] LustreError: 18686:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760452079 with bad export cookie 5601075482732421347 [ 332.975744] LustreError: 166-1: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 332.977523] LustreError: 18686:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 340.960156] Lustre: 21833:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760452081/real 1760452081] req@0000000027297315 x1845967532918720/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1760452087 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 350.007143] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 354.675267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 354.874399] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 354.908767] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 354.912491] Lustre: Skipped 2 previous similar messages [ 356.841116] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 360.910684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 363.024919] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 366.952326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 369.107357] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 373.525537] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 373.560851] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 373.565371] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 373.649281] Lustre: lustre-OST0001: new disk, initializing [ 373.651525] Lustre: Skipped 1 previous similar message [ 373.655638] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 373.659525] Lustre: Skipped 1 previous similar message [ 375.142406] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 375.146825] Lustre: Skipped 1 previous similar message [ 375.150576] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 375.765909] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 381.380877] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 383.248882] Lustre: Setting parameter general.lod.*.mdt_hash in log params [ 396.773063] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 396.775669] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 396.779958] Lustre: Skipped 3 previous similar messages [ 396.782014] Lustre: Skipped 6 previous similar messages [ 398.817434] Lustre: server umount lustre-MDT0000 complete [ 398.819289] Lustre: Skipped 2 previous similar messages [ 400.522170] LustreError: 23246:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760452147 with bad export cookie 5601075482732422194 [ 400.523309] LustreError: 166-1: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 400.527325] LustreError: 23246:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 408.544193] Lustre: 27028:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760452149/real 1760452149] req@00000000fb22c65f x1845967532944832/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1760452155 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 418.119100] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 423.840780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 424.052503] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 424.064248] LustreError: Skipped 1 previous similar message [ 424.116240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 424.122104] Lustre: Skipped 3 previous similar messages [ 426.499482] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 430.961744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 433.529557] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 437.926825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 440.406256] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 445.345742] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 445.414768] Lustre: lustre-OST0001: Not available for connect from 0@lo (not set up) [ 448.039954] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 452.280574] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 453.925628] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 453.927848] Lustre: Skipped 1 previous similar message [ 461.794113] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 461.795043] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 461.802755] Lustre: Skipped 3 previous similar messages [ 467.214379] Lustre: server umount lustre-MDT0000 complete [ 467.216149] Lustre: Skipped 3 previous similar messages [ 468.768148] LustreError: 28438:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760452215 with bad export cookie 5601075482732423440 [ 468.770196] LustreError: 166-1: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 468.773516] LustreError: 28438:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 477.152181] Lustre: 32061:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760452217/real 1760452217] req@00000000ed0aac1b x1845967532965056/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1760452223 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 477.162930] Lustre: 32061:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 484.832172] Lustre: 32262:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760452225/real 1760452225] req@00000000211c841e x1845967532965568/t0(0) o39->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1760452231 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 488.534945] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 10:30:34 (1760452234) [ 499.223766] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 503.025630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 503.181079] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 503.188601] LustreError: Skipped 2 previous similar messages [ 504.841744] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 508.196445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 509.999555] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 513.369993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 515.138959] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 518.425116] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 520.177713] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 523.848945] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 525.297621] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 533.987301] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 533.989254] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 533.990195] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 533.990204] Lustre: Skipped 8 previous similar messages [ 534.000148] Lustre: Skipped 3 previous similar messages [ 537.869373] Lustre: server umount lustre-MDT0000 complete [ 537.871264] Lustre: Skipped 3 previous similar messages [ 539.304050] LustreError: 35877:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760452285 with bad export cookie 5601075482732424511 [ 539.306496] LustreError: 166-1: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 539.308601] LustreError: 35877:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 547.296224] Lustre: 37850:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760452287/real 1760452287] req@00000000fdeb58c4 x1845967532984128/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1760452293 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 558.358706] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 10:31:44 (1760452304) [ 564.130360] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 567.618328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 567.766806] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 567.773899] LustreError: Skipped 7 previous similar messages [ 567.786616] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 567.789328] Lustre: Skipped 7 previous similar messages [ 569.261648] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 572.474152] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 574.336579] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 577.554408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 579.316326] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 582.527242] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 584.247204] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 587.607410] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 598.496852] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 602.544830] LustreError: 41057:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760452349 with bad export cookie 5601075482732425582 [ 602.547808] LustreError: 166-1: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 602.548747] LustreError: 41057:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 611.845423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 619.489414] Lustre: Evicted from MGS (at 192.168.203.149@tcp) after server handle changed from 0x4dbb0479cc6a716e to 0x4dbb0479cc6a75e3 [ 619.493805] Lustre: MGC192.168.203.149@tcp: Connection restored to (at 0@lo) [ 619.639262] BUG: spinlock bad magic on CPU#0, lctl/43387 [ 619.641280] lock: 0xffff8d457022e500, .magic: 00000000, .owner: /-1, .owner_cpu: 0 [ 619.644733] CPU: 0 PID: 43387 Comm: lctl Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 [ 619.647464] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 619.650486] Call Trace: [ 619.652188] ? dump_stack+0xbb/0x10e [ 619.653489] ? spin_dump.cold.3+0x32/0x65 [ 619.654782] ? do_raw_spin_lock+0xce/0x190 [ 619.656114] ? _raw_spin_lock+0x21/0x30 [ 619.657439] ? mdd_changelog_mask_seq_write+0x61a/0x660 [mdd] [ 619.659491] ? __seccomp_filter+0x4b/0x680 [ 619.660868] ? full_proxy_write+0x5e/0xa0 [ 619.662567] ? __vfs_write+0x1c/0x60 [ 619.664111] ? vfs_write+0xd8/0x2b0 [ 619.665007] ? ksys_write+0x66/0x120 [ 619.666558] ? __x64_sys_write+0x1e/0x30 [ 619.668322] ? do_syscall_64+0xc1/0x440 [ 619.669807] ? entry_SYSCALL_64_after_hwframe+0x49/0xae [ 621.268067] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 624.533794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 625.701600] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to (at 0@lo) [ 626.276175] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 628.960029] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 629.796562] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to (at 0@lo) [ 629.798892] Lustre: Skipped 1 previous similar message [ 633.600965] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 635.646834] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 636.247598] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 639.973820] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 645.201785] LustreError: 40279:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760452391 with bad export cookie 5601075482732426723 [ 645.205813] LustreError: 40279:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 645.206592] LustreError: 166-1: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 652.768339] Lustre: 45601:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760452393/real 1760452393] req@000000007347300c x1845967533018368/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1760452399 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 652.777642] Lustre: 45601:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 662.819582] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 10:33:28 (1760452408) [ 668.499081] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 672.002806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 673.780373] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 676.969926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 678.781815] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 681.968085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 683.610698] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 686.638888] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 688.159453] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 691.257725] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 692.485236] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 692.487147] Lustre: Skipped 2 previous similar messages [ 702.944758] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 702.947236] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 702.951287] Lustre: Skipped 11 previous similar messages [ 702.953372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 702.955219] Lustre: Skipped 19 previous similar messages [ 708.042983] Lustre: server umount lustre-MDT0000 complete [ 708.044745] Lustre: Skipped 9 previous similar messages [ 708.578076] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 708.583294] LustreError: Skipped 17 previous similar messages [ 709.315981] LustreError: 47213:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760452455 with bad export cookie 5601075482732427507 [ 709.318820] LustreError: 166-1: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 709.320034] LustreError: 47213:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 717.280152] Lustre: 50798:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760452457/real 1760452457] req@000000006050741d x1845967533040000/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1760452463 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 717.287963] Lustre: 50798:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 727.815315] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 10:34:34 (1760452474) [ 728.683597] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 729.297865] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 10:34:35 (1760452475) [ 732.974613] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_hostid [ 735.658185] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 738.941814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 740.953957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 743.232917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 743.253859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 743.345468] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 743.383109] Lustre: lustre-MDT0000: new disk, initializing [ 743.412627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 744.865677] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 748.757124] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 750.994066] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 751.017173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 751.020258] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 752.535320] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 756.376515] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 757.098710] Lustre: DEBUG MARKER: create test files [ 768.481463] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 779.744203] Lustre: 55487:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760452520/real 1760452520] req@00000000ed9c634c x1845967533047168/t0(0) o251->MGC192.168.203.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1760452526 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 779.751467] Lustre: 55487:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 784.238100] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 785.217240] Key type lgssc unregistered [ 785.316307] LNet: 56089:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 785.318730] LNet: Removed LNI 192.168.203.149@tcp [ 785.622412] Key type .llcrypt unregistered [ 785.623698] Key type ._llcrypt unregistered [ 795.719429] alg: No test for adler32 (adler32-zlib) [ 796.471384] Key type ._llcrypt registered [ 796.472271] Key type .llcrypt registered [ 796.515701] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 796.979530] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 797.251556] Lustre: Lustre: Build Version: 2.15.7_5_g4a1d30d [ 797.324900] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 797.326442] LNet: Accept secure, port 988 [ 798.904101] Key type lgssc registered [ 799.235680] Lustre: Echo OBD driver; http://www.lustre.org/ [ 801.854721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 803.016128] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 804.475030] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 806.900174] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 809.261075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 809.364956] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 810.765846] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 813.214695] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 818.660399] Lustre: lustre-OST0000: deleting orphan objects from 0x0:102 to 0x0:129 [ 818.682414] Lustre: 58942:0:(osd_internal.h:1344:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 515 < left 544, rollback = 9 [ 818.685298] Lustre: 58942:0:(osd_handler.c:1944:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 818.687503] Lustre: 58942:0:(osd_handler.c:1951:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 818.689877] Lustre: 58942:0:(osd_handler.c:1961:osd_trans_dump_creds()) write: 1/1/1, punch: 0/0/0, quota 1/3/0 [ 818.692358] Lustre: 58942:0:(osd_handler.c:1968:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 818.695402] Lustre: 58942:0:(osd_handler.c:1975:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 826.847459] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 828.449296] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 834.017337] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 834.017751] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 834.026742] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 836.372799] Lustre: server umount lustre-OST0000 complete [ 843.744225] Lustre: 59977:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760452584/real 1760452584] req@00000000369aea6a x1845968285734208/t0(0) o251->MGC192.168.203.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1760452590 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 843.808247] Lustre: server umount lustre-MDT0000 complete [ 848.237670] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 849.192184] Key type lgssc unregistered [ 849.307389] LNet: 60582:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 849.310145] LNet: Removed LNI 192.168.203.149@tcp [ 849.580422] Key type .llcrypt unregistered [ 849.582177] Key type ._llcrypt unregistered [ 860.356269] alg: No test for adler32 (adler32-zlib) [ 861.108386] Key type ._llcrypt registered [ 861.109809] Key type .llcrypt registered [ 861.148059] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 861.628547] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 861.758852] Lustre: Lustre: Build Version: 2.15.7_5_g4a1d30d [ 861.844968] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 861.847084] LNet: Accept secure, port 988 [ 863.440127] Key type lgssc registered [ 863.794820] Lustre: Echo OBD driver; http://www.lustre.org/ [ 866.303218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 867.440262] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 868.701190] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 871.111890] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 873.182827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 873.272474] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 874.558665] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 876.969425] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 882.660431] Lustre: lustre-OST0000: deleting orphan objects from 0x0:140 to 0x0:161 [ 882.678248] Lustre: 63515:0:(osd_internal.h:1344:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 515 < left 544, rollback = 9 [ 882.681133] Lustre: 63515:0:(osd_handler.c:1944:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 882.683154] Lustre: 63515:0:(osd_handler.c:1951:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 882.686290] Lustre: 63515:0:(osd_handler.c:1961:osd_trans_dump_creds()) write: 1/1/1, punch: 0/0/0, quota 1/3/0 [ 882.688864] Lustre: 63515:0:(osd_handler.c:1968:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 882.691647] Lustre: 63515:0:(osd_handler.c:1975:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 890.756088] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 892.897138] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 892.897195] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 892.901444] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 898.016873] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 898.838098] Lustre: server umount lustre-OST0000 complete [ 906.208182] Lustre: 64501:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760452646/real 1760452646] req@000000008cbaaf22 x1845968353891520/t0(0) o251->MGC192.168.203.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1760452652 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 906.276352] Lustre: server umount lustre-MDT0000 complete [ 910.686973] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 911.773164] Key type lgssc unregistered [ 911.882361] LNet: 65105:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 911.885412] LNet: Removed LNI 192.168.203.149@tcp [ 912.195261] Key type .llcrypt unregistered [ 912.196440] Key type ._llcrypt unregistered [ 918.533758] alg: No test for adler32 (adler32-zlib) [ 919.285381] Key type ._llcrypt registered [ 919.287109] Key type .llcrypt registered [ 919.326107] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_hostid [ 925.277314] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 925.815098] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 926.000419] Lustre: Lustre: Build Version: 2.15.7_5_g4a1d30d [ 926.094450] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 926.096162] LNet: Accept secure, port 988 [ 927.688154] Key type lgssc registered [ 928.102468] Lustre: Echo OBD driver; http://www.lustre.org/ [ 931.026843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 933.067496] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 934.950412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 936.906311] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 939.039969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 939.061927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 940.137717] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 940.147229] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 940.180522] Lustre: lustre-MDT0000: new disk, initializing [ 940.202259] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 940.208996] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 941.501583] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 945.436265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 945.458116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 945.488406] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 945.502280] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 945.503894] Lustre: Skipped 1 previous similar message [ 945.537521] Lustre: lustre-MDT0001: new disk, initializing [ 945.564429] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 945.574596] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 945.578041] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 946.895572] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 950.580434] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 952.400130] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 954.688583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 954.720856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 954.723436] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 954.811247] Lustre: lustre-OST0000: new disk, initializing [ 954.813896] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 954.838524] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 956.147580] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 959.856968] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 961.274284] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 963.060698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 963.064167] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 963.391168] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 964.799617] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 964.882095] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 968.162436] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 968.162448] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 968.163691] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 968.167088] Lustre: Skipped 1 previous similar message [ 968.170253] LustreError: Skipped 1 previous similar message [ 971.670703] Lustre: server umount lustre-OST0000 complete [ 973.281324] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 973.281718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 973.286122] Lustre: Skipped 1 previous similar message [ 973.288359] Lustre: Skipped 1 previous similar message [ 978.401340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 978.404029] Lustre: Skipped 2 previous similar messages [ 979.082409] Lustre: server umount lustre-MDT0000 complete [ 980.336391] LustreError: 69226:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760452726 with bad export cookie 1609116684270188530 [ 980.338493] LustreError: 166-1: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 980.341266] LustreError: 69226:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 980.455762] Lustre: server umount lustre-MDT0001 complete [ 983.283592] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 10:38:49 (1760452729) [ 986.704307] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 991.286126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 999.382983] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_hostid [ 1002.337496] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 1005.495133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1007.664297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1009.622606] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1011.661529] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1013.944812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1013.968378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1014.048175] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1014.058577] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1014.093543] Lustre: lustre-MDT0000: new disk, initializing [ 1014.115880] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1014.121824] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1015.503083] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 1019.630175] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1019.655464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1019.691524] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1019.709231] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1019.710772] Lustre: Skipped 1 previous similar message [ 1019.752611] Lustre: lustre-MDT0001: new disk, initializing [ 1019.785926] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1019.799427] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1019.803041] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1021.351391] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 1025.365888] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1027.342913] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1029.598029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1029.622839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1029.625140] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1029.706191] Lustre: lustre-OST0000: new disk, initializing [ 1029.707890] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1029.737123] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1031.223864] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 1034.954592] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1036.360237] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1036.791931] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1036.795277] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1037.456380] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 1038.803897] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 1038.871087] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1041.889370] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1041.891128] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1041.893084] LustreError: Skipped 1 previous similar message [ 1041.897220] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1041.899837] Lustre: Skipped 1 previous similar message [ 1045.588197] Lustre: server umount lustre-OST0000 complete [ 1047.009801] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1047.010476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1047.014131] Lustre: Skipped 2 previous similar messages [ 1057.249293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1057.251211] Lustre: Skipped 4 previous similar messages [ 1061.344101] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1061.380371] Lustre: server umount lustre-MDT0000 complete [ 1062.705318] LustreError: 77526:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760452809 with bad export cookie 1609116684270189370 [ 1062.707679] LustreError: 166-1: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1062.708964] LustreError: 77526:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1065.488858] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 10:40:11 (1760452811) [ 1067.633193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1067.780544] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1067.800420] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1069.175499] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 1072.021315] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1073.642669] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 1076.242291] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1078.159713] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1080.482200] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1082.085205] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 1084.513878] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1085.728514] Lustre: 83968:0:(genops.c:1710:obd_export_evict_by_uuid()) MGS: evicting d28ddab1-1ed9-48bc-b886-f90e010d6103 at adminstrative request [ 1086.753589] LustreError: 166-1: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1086.759793] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 1093.088934] Lustre: MGS: Client d28ddab1-1ed9-48bc-b886-f90e010d6103 (at 0@lo) reconnecting [ 1093.092548] Lustre: Evicted from MGS (at 192.168.203.149@tcp) after server handle changed from 0x1654bb1950143e82 to 0x1654bb1950144114 [ 1093.096419] Lustre: MGC192.168.203.149@tcp: Connection restored to 192.168.203.149@tcp (at 0@lo) [ 1117.998324] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1118.023411] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1118.025956] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 1118.070371] Lustre: lustre-OST0001: new disk, initializing [ 1118.072226] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 1118.098218] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 1118.100525] Lustre: Skipped 2 previous similar messages [ 1119.397320] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 1123.172815] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 1124.170803] Lustre: server umount lustre-OST0001 complete [ 1124.172080] Lustre: Skipped 1 previous similar message [ 1125.870273] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1125.871577] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1125.880022] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1125.882572] Lustre: Skipped 3 previous similar messages [ 1130.978881] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1130.983138] LustreError: Skipped 2 previous similar messages [ 1133.536685] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1136.097884] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1136.102302] Lustre: Skipped 2 previous similar messages [ 1147.360181] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1147.408930] Lustre: server umount lustre-MDT0000 complete [ 1147.410800] Lustre: Skipped 1 previous similar message [ 1148.744708] LustreError: 84362:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760452895 with bad export cookie 1609116684270190868 [ 1148.747547] LustreError: 166-1: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1148.749310] LustreError: 84362:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1151.691473] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 10:41:37 (1760452897) [ 1152.211188] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 1152.769822] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 10:41:38 (1760452898) [ 1153.308157] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 1153.960679] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 10:41:40 (1760452900) [ 1154.518285] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 1155.077113] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 10:41:41 (1760452901) [ 1155.575823] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 1157.184514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1157.528990] loop: module loaded [ 1157.532157] loop0: detected capacity change from 0 to 4294967296 [ 1157.542697] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 1158.031398] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 1159.126854] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 1160.476748] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 1163.511186] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1164.849688] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1166.058885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1167.331682] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1169.494617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1169.521366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1169.524252] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1169.529261] LustreError: 88404:0:(obd_config.c:774:class_setup()) setup lustre-OST0000-osd failed (-28) [ 1169.532485] LustreError: 88404:0:(obd_mount.c:200:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 1169.536166] LustreError: 88404:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 1169.538832] LustreError: 88404:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 1175.061478] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_hostid [ 1177.894603] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 1181.096210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1183.160483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1185.062736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1187.036987] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1189.174823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1189.196865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1189.268066] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1189.277172] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1189.312451] Lustre: lustre-MDT0000: new disk, initializing [ 1189.334362] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1189.341092] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1190.693359] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 1194.609605] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1194.631405] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1194.745592] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1196.069110] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 1199.771923] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1201.809688] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1203.957420] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1203.982015] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1203.984375] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1205.419399] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 1209.186941] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1209.318935] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1210.575353] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1210.641586] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1211.991142] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 1212.062233] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1214.434324] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1214.434934] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1214.435362] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1214.435367] Lustre: Skipped 9 previous similar messages [ 1214.437375] LustreError: Skipped 1 previous similar message [ 1214.440449] Lustre: Skipped 1 previous similar message [ 1218.770233] Lustre: server umount lustre-OST0000 complete [ 1218.771567] Lustre: Skipped 1 previous similar message [ 1220.576644] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1234.400176] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1235.803476] LustreError: 94936:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760452982 with bad export cookie 1609116684270191407 [ 1235.807507] LustreError: 166-1: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1235.808721] LustreError: 94936:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1238.931784] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 10:43:05 (1760452985) [ 1241.295165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1241.457124] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1243.024616] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 1246.040319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1247.701866] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 1250.356049] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1252.291726] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1254.464843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1255.868080] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 1258.171228] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1260.325530] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1260.347239] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1260.350088] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 1260.389604] Lustre: lustre-OST0001: new disk, initializing [ 1260.390761] Lustre: Skipped 2 previous similar messages [ 1260.392295] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 1260.394848] Lustre: Skipped 3 previous similar messages [ 1261.707188] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 1265.368689] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 1266.683530] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1266.685576] Lustre: Skipped 2 previous similar messages [ 1266.686979] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1274.938887] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1278.652246] Lustre: Failing over lustre-MDT0000 [ 1281.975615] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1281.980644] LustreError: Skipped 1 previous similar message [ 1282.018592] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1282.020570] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1282.021989] Lustre: Skipped 2 previous similar messages [ 1289.184206] Lustre: 66603:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760453028/real 1760453028] req@00000000f8263bf3 x1845968421085632/t0(0) o400->MGC192.168.203.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1760453035 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1289.191182] LustreError: 166-1: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1292.208860] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1292.213132] LustreError: Skipped 17 previous similar messages [ 1293.818673] LDISKFS-fs (dm-0): recovery complete [ 1293.820387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1295.332597] Lustre: Evicted from MGS (at 192.168.203.149@tcp) after server handle changed from 0x1654bb1950144677 to 0x1654bb195014d44c [ 1295.339045] Lustre: MGC192.168.203.149@tcp: Connection restored to (at 0@lo) [ 1295.438806] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1296.772631] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 1299.312662] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 1299.312795] Lustre: 95801:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 1300.450615] Lustre: 96727:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 1300.454806] Lustre: 96727:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 1 previous similar message [ 1300.459644] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 1300.482995] LustreError: 100964:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 1300.800124] LustreError: 100964:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 1301.114580] LustreError: 100964:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 1301.117406] LustreError: 100964:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 1 previous similar message [ 1301.432114] LustreError: 100964:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 1301.435127] LustreError: 100964:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 1 previous similar message [ 1301.753691] Lustre: 100964:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1301.756993] Lustre: 100964:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 8 previous similar messages [ 1302.385589] LustreError: 100964:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 1302.388726] LustreError: 100964:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 3 previous similar messages [ 1302.704158] LustreError: 100964:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 1302.706425] LustreError: 100964:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 3 previous similar messages [ 1303.960715] Lustre: 100964:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1303.963891] Lustre: 100964:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 6 previous similar messages [ 1304.592780] LustreError: 100964:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 1304.595223] LustreError: 100964:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 6 previous similar messages [ 1304.912119] LustreError: 100964:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 1304.914157] LustreError: 100964:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 6 previous similar messages [ 1308.041784] Lustre: 100964:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1308.046375] Lustre: 100964:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 12 previous similar messages [ 1308.674300] LustreError: 100964:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 1308.678324] LustreError: 100964:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 12 previous similar messages [ 1308.992144] LustreError: 100964:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 1308.994855] LustreError: 100964:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 12 previous similar messages [ 1316.177854] Lustre: 100964:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1316.182114] Lustre: 100964:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 25 previous similar messages [ 1316.809839] LustreError: 100964:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 1316.813970] LustreError: 100964:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 25 previous similar messages [ 1317.128113] LustreError: 100964:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 1317.130413] LustreError: 100964:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 25 previous similar messages [ 1332.426110] Lustre: 100964:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1332.430118] Lustre: 100964:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 51 previous similar messages [ 1333.057982] LustreError: 100964:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 1333.061786] LustreError: 100964:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 51 previous similar messages [ 1333.376128] LustreError: 100964:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 1333.379830] LustreError: 100964:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 51 previous similar messages [ 1359.280151] Lustre: 100964:0:(ldlm_lib.c:2278:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 1359.283374] Lustre: 100964:0:(ldlm_lib.c:2288:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1359.286662] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 1359.288030] LustreError: 100964:0:(tgt_grant.c:257:tgt_grant_sanity_check()) mdt_obd_disconnect: tot_granted 2097152 != fo_tot_granted 4194304 [ 1359.291728] Lustre: 100964:0:(ldlm_lib.c:1803:abort_req_replay_queue()) @@@ aborted: req@0000000040dc1d58 x1845968415781184/t0(8589934785) o101->6d2eef3a-2ce9-49b6-82af-0f4ad6595fa6@192.168.203.49@tcp:736/0 lens 592/0 e 0 to 0 dl 1760453111 ref 1 fl Complete:/4/ffffffff rc 0/-1 job:'' [ 1359.300345] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to (at 0@lo) [ 1359.303302] Lustre: Skipped 2 previous similar messages [ 1359.303756] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 1359.323275] Lustre: lustre-OST0001: deleting orphan objects from 0x0:49 to 0x0:545 [ 1359.323328] Lustre: lustre-OST0000: deleting orphan objects from 0x0:81 to 0x0:577 [ 1363.937090] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1363.940243] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1363.944535] Lustre: Skipped 3 previous similar messages [ 1363.946364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1363.948443] Lustre: Skipped 8 previous similar messages [ 1368.364948] Lustre: server umount lustre-OST0000 complete [ 1368.367225] Lustre: Skipped 3 previous similar messages [ 1372.130086] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1372.135798] LustreError: Skipped 13 previous similar messages [ 1384.380795] LustreError: 97758:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760453130 with bad export cookie 1609116684270228556 [ 1384.383548] LustreError: 166-1: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1384.384753] LustreError: 97758:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1387.230104] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 10:45:33 (1760453133) [ 1389.667187] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1389.669307] Lustre: *** cfs_fail_loc=197, val=0*** [ 1406.112595] LustreError: 102987:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.203.149@tcp: failed processing log, type 1: rc = -5 [ 1417.376529] LustreError: 102987:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.203.149@tcp: failed processing log, type 4: rc = -110 [ 1423.526836] Lustre: *** cfs_fail_loc=197, val=0*** [ 1423.528512] Lustre: Skipped 65 previous similar messages [ 1446.048289] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1446.050453] Lustre: Skipped 7 previous similar messages [ 1447.533053] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 1450.092394] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1453.455919] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 10:46:39 (1760453199) [ 1467.438868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1467.585213] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1469.097938] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 1472.923214] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1474.920884] LustreError: 104641:0:(osp_object.c:629:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1474.923820] LustreError: 104641:0:(lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1474.926451] LustreError: 104641:0:(lod_dev.c:424:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000 get update log failed: rc = -5 [ 1481.184161] Lustre: 105332:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760453221/real 1760453221] req@0000000083e845f6 x1845968421113728/t0(0) o251->MGC192.168.203.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1760453227 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1486.448270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1492.131442] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_hostid [ 1495.062253] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 1498.625778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1500.914351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1502.980794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1505.182106] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1507.522203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1507.543360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1507.624185] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1507.626398] Lustre: Skipped 1 previous similar message [ 1507.635963] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1507.677357] Lustre: lustre-MDT0000: new disk, initializing [ 1507.705473] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1509.162305] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 1513.445565] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1513.465730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1513.589093] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1515.049031] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 1518.952499] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1520.875162] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1523.091857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1523.116464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1523.119487] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1524.663742] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 1528.545145] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1530.136481] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1530.211913] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1531.705320] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 1531.784922] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1533.922166] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1533.922197] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1533.924837] LustreError: Skipped 3 previous similar messages [ 1533.928410] Lustre: Skipped 5 previous similar messages [ 1533.928935] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1533.934183] Lustre: Skipped 11 previous similar messages [ 1538.520152] Lustre: server umount lustre-OST0000 complete [ 1538.522728] Lustre: Skipped 5 previous similar messages [ 1554.400114] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1555.426068] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1555.432185] LustreError: Skipped 2 previous similar messages [ 1555.708036] LustreError: 109809:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760453302 with bad export cookie 1609116684270239280 [ 1555.710788] LustreError: 166-1: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1555.713416] LustreError: 109809:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1558.731922] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 10:48:24 (1760453304) [ 1562.835608] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 1563.976667] Key type lgssc unregistered [ 1564.088353] LNet: 113634:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1564.090921] LNet: Removed LNI 192.168.203.149@tcp [ 1564.397888] Key type .llcrypt unregistered [ 1564.399121] Key type ._llcrypt unregistered [ 1570.862022] alg: No test for adler32 (adler32-zlib) [ 1571.613482] Key type ._llcrypt registered [ 1571.615067] Key type .llcrypt registered [ 1571.656229] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_hostid [ 1577.295548] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 1577.663744] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1577.912488] Lustre: Lustre: Build Version: 2.15.7_5_g4a1d30d [ 1577.999648] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 1578.001392] LNet: Accept secure, port 988 [ 1579.576104] Key type lgssc registered [ 1579.915596] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1582.970295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1585.269834] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1587.406056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1589.602212] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1591.514440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1593.895201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1593.918745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1595.005687] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1595.017079] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1595.052374] Lustre: lustre-MDT0000: new disk, initializing [ 1595.076214] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1595.085132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1596.593601] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 1600.714238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1600.745154] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1600.749215] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1600.835067] Lustre: lustre-OST0000: new disk, initializing [ 1600.837157] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1600.839314] Lustre: Skipped 1 previous similar message [ 1600.859543] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1602.342315] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 1606.418592] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1606.442477] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1606.446615] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 1606.500082] Lustre: lustre-OST0001: new disk, initializing [ 1606.501966] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 1606.525912] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 1607.958582] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 1615.329152] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1615.329920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1615.332502] Lustre: Skipped 1 previous similar message [ 1618.171521] Lustre: server umount lustre-MDT0000 complete [ 1620.340555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1621.916272] LustreError: 117130:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760453368 with bad export cookie 2150737048178865328 [ 1621.918177] LustreError: 166-1: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1621.920238] LustreError: 117130:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1621.931626] Lustre: server umount lustre-OST0000 complete [ 1623.299381] Lustre: server umount lustre-OST0001 complete [ 1626.260399] Lustre: DEBUG MARKER: == conf-sanity test 88: check the default mount options can be overridden ========================================================== 10:49:32 (1760453372) [ 1629.270505] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 1632.466843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1632.828831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1634.692182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1635.034508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1637.257336] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 10:49:43 (1760453383) [ 1641.686418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1642.372474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1643.043189] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1643.704162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1644.074123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1644.794866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1650.788995] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_hostid [ 1653.920961] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 1657.239980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1659.475979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1661.338626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1663.455040] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1665.823454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1665.847965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1665.924794] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1665.934900] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1665.971608] Lustre: lustre-MDT0000: new disk, initializing [ 1665.996437] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1666.002841] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1667.415540] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 1671.535977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1671.561751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1671.595891] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1671.611526] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1671.613493] Lustre: Skipped 1 previous similar message [ 1671.653067] Lustre: lustre-MDT0001: new disk, initializing [ 1671.686431] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1671.700084] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1671.704902] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1673.137355] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 1676.921502] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1678.921756] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1681.143223] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1681.170667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1681.172985] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1681.257547] Lustre: lustre-OST0000: new disk, initializing [ 1681.259712] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1681.286422] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1682.670344] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 1686.554434] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1688.018217] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1691.127743] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1691.131519] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1692.177438] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 1693.638175] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 1693.722333] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1696.225796] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1696.226899] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1696.228567] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1696.231407] Lustre: Skipped 1 previous similar message [ 1696.233490] Lustre: Skipped 1 previous similar message [ 1700.503665] Lustre: server umount lustre-OST0000 complete [ 1702.368773] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1702.371203] LustreError: Skipped 1 previous similar message [ 1702.372185] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1702.375323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1702.377061] Lustre: Skipped 1 previous similar message [ 1706.464920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1706.465452] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1711.584915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1711.587091] Lustre: Skipped 2 previous similar messages [ 1716.192136] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1716.232485] Lustre: server umount lustre-MDT0000 complete [ 1716.704900] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1716.708288] LustreError: Skipped 1 previous similar message [ 1717.507067] LustreError: 126510:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760453464 with bad export cookie 2150737048178866518 [ 1717.509896] LustreError: 166-1: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1717.510549] LustreError: 126510:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1720.637548] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 10:51:06 (1760453466) [ 1722.872051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1723.031387] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1723.054648] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1724.566629] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 1727.582715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1729.190308] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 1731.853894] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1733.798893] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1736.130629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1737.760801] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 1740.476254] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1746.397421] Lustre: *** cfs_fail_loc=159, val=0*** [ 1746.399165] Lustre: Skipped 1 previous similar message [ 1753.534038] Lustre: lustre-MDT0000: Client c72a0e1e-c94a-4252-a586-cf99424a50db (at 192.168.203.49@tcp) reconnecting [ 1753.888931] Lustre: *** cfs_fail_loc=159, val=0*** [ 1753.890537] Lustre: Skipped 3 previous similar messages [ 1761.214139] Lustre: lustre-MDT0000: Client c72a0e1e-c94a-4252-a586-cf99424a50db (at 192.168.203.49@tcp) reconnecting [ 1763.810572] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1763.810620] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1763.811667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1763.814613] Lustre: Skipped 1 previous similar message [ 1768.349833] Lustre: server umount lustre-OST0000 complete [ 1768.351257] Lustre: Skipped 1 previous similar message [ 1774.049608] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1774.049661] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1774.055333] Lustre: Skipped 1 previous similar message [ 1780.193140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1780.195209] Lustre: Skipped 6 previous similar messages [ 1784.288072] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1785.313155] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1785.317112] LustreError: Skipped 1 previous similar message [ 1785.522185] LustreError: 130218:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760453532 with bad export cookie 2150737048178867358 [ 1785.524441] LustreError: 166-1: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1785.524901] LustreError: 130218:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1785.656052] Lustre: server umount lustre-MDT0001 complete [ 1785.657423] Lustre: Skipped 1 previous similar message [ 1790.208232] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 1791.224467] Key type lgssc unregistered [ 1791.339410] LNet: 134088:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1791.343568] LNet: Removed LNI 192.168.203.149@tcp [ 1791.644289] Key type .llcrypt unregistered [ 1791.645478] Key type ._llcrypt unregistered [ 1800.164115] alg: No test for adler32 (adler32-zlib) [ 1800.916300] Key type ._llcrypt registered [ 1800.917707] Key type .llcrypt registered [ 1800.952450] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 1801.307766] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1801.591337] Lustre: Lustre: Build Version: 2.15.7_5_g4a1d30d [ 1801.679807] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 1801.682188] LNet: Accept secure, port 988 [ 1803.264126] Key type lgssc registered [ 1803.589304] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1806.073452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1807.196555] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1807.209635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1808.485517] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 1811.168344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1811.281412] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1812.673043] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 1815.040313] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1816.776196] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1818.869347] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1818.989069] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1820.364139] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 1822.054449] Lustre: lustre-OST0000: deleting orphan objects from 0x0:42 to 0x0:65 [ 1822.742545] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1832.335458] Lustre: *** cfs_fail_loc=159, val=0*** [ 1839.550064] Lustre: lustre-MDT0000: Client 7f90ebb7-9ab0-4edb-90a7-eb2db9f5ab8f (at 192.168.203.49@tcp) reconnecting [ 1840.023919] Lustre: *** cfs_fail_loc=159, val=0*** [ 1840.025772] Lustre: Skipped 1 previous similar message [ 1847.233297] Lustre: lustre-MDT0001: Client 7f90ebb7-9ab0-4edb-90a7-eb2db9f5ab8f (at 192.168.203.49@tcp) reconnecting [ 1847.553583] Lustre: *** cfs_fail_loc=159, val=0*** [ 1847.554940] Lustre: Skipped 2 previous similar messages [ 1854.909559] Lustre: lustre-MDT0001: Client 7f90ebb7-9ab0-4edb-90a7-eb2db9f5ab8f (at 192.168.203.49@tcp) reconnecting [ 1856.441902] Lustre: *** cfs_fail_loc=159, val=0*** [ 1856.443336] Lustre: Skipped 5 previous similar messages [ 1863.613970] Lustre: lustre-MDT0000: Client 3c95aeb0-098c-4667-8587-5b781fb29a6c (at 192.168.203.49@tcp) reconnecting [ 1863.985796] Lustre: *** cfs_fail_loc=159, val=0*** [ 1863.987663] Lustre: Skipped 13 previous similar messages [ 1871.292369] Lustre: lustre-MDT0000: Client 3c95aeb0-098c-4667-8587-5b781fb29a6c (at 192.168.203.49@tcp) reconnecting [ 1873.376985] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1873.380708] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1873.386715] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1877.473780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1877.473794] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1877.477022] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1877.479648] LustreError: Skipped 1 previous similar message [ 1879.073372] Lustre: server umount lustre-OST0000 complete [ 1882.594308] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1882.595212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1882.599028] Lustre: Skipped 1 previous similar message [ 1882.603015] Lustre: Skipped 2 previous similar messages [ 1886.599996] Lustre: server umount lustre-MDT0000 complete [ 1887.713788] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1887.718841] LustreError: Skipped 1 previous similar message [ 1887.888061] LustreError: 135528:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760453634 with bad export cookie 16518452295433082247 [ 1887.890887] LustreError: 166-1: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1887.892227] LustreError: 135528:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1888.016056] Lustre: server umount lustre-MDT0001 complete [ 1892.729371] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 1893.652959] Key type lgssc unregistered [ 1893.780462] LNet: 139937:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1893.782761] LNet: Removed LNI 192.168.203.149@tcp [ 1894.086038] Key type .llcrypt unregistered [ 1894.086956] Key type ._llcrypt unregistered [ 1902.787069] alg: No test for adler32 (adler32-zlib) [ 1903.538379] Key type ._llcrypt registered [ 1903.539393] Key type .llcrypt registered [ 1903.572592] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 1904.117314] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1904.276854] Lustre: Lustre: Build Version: 2.15.7_5_g4a1d30d [ 1904.362610] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 1904.364329] LNet: Accept secure, port 988 [ 1905.944164] Key type lgssc registered [ 1906.339289] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1909.015865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1910.175303] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1910.191609] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1911.613322] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 1914.603864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1914.717060] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1916.187615] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 1918.669326] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1920.613738] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1922.976274] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1923.105398] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1924.660733] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 1926.181524] Lustre: lustre-OST0000: deleting orphan objects from 0x0:85 to 0x0:129 [ 1927.378875] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1930.280134] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:72 to 0x280000400:97 [ 1935.596762] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight in log params [ 1936.771445] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight in log params [ 1936.774224] Lustre: Skipped 1 previous similar message [ 1940.450642] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1940.452264] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1940.454579] Lustre: Skipped 1 previous similar message [ 1940.454774] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1940.456914] Lustre: Skipped 1 previous similar message [ 1944.554934] Lustre: server umount lustre-OST0000 complete [ 1950.689875] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1950.690093] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1950.690817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1950.698374] Lustre: Skipped 1 previous similar message [ 1952.012429] Lustre: server umount lustre-MDT0000 complete [ 1953.350084] LustreError: 144065:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760453699 with bad export cookie 3920809748981948028 [ 1953.352520] LustreError: 166-1: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1953.353128] LustreError: 144065:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1953.502591] Lustre: server umount lustre-MDT0001 complete [ 1958.455972] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 1959.623739] Key type lgssc unregistered [ 1959.733329] LNet: 145458:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1959.735981] LNet: Removed LNI 192.168.203.149@tcp [ 1960.031254] Key type .llcrypt unregistered [ 1960.032438] Key type ._llcrypt unregistered [ 1969.478365] alg: No test for adler32 (adler32-zlib) [ 1970.229381] Key type ._llcrypt registered [ 1970.231016] Key type .llcrypt registered [ 1970.270607] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 1970.706694] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1970.868366] Lustre: Lustre: Build Version: 2.15.7_5_g4a1d30d [ 1970.948528] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 1970.951279] LNet: Accept secure, port 988 [ 1972.536100] Key type lgssc registered [ 1972.926414] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1975.958995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1977.137050] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1977.153765] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1978.777715] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 1981.869907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1982.015303] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1983.612864] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 1986.414127] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1988.402635] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1990.651620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1990.767474] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1992.149496] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 1992.804392] Lustre: lustre-OST0000: deleting orphan objects from 0x0:85 to 0x0:161 [ 1994.619853] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1996.901617] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:72 to 0x280000400:129 [ 2001.367425] Lustre: *** cfs_fail_loc=159, val=0*** [ 2008.509612] Lustre: lustre-MDT0000: Client a1906607-474b-4113-ad6b-246b48983bac (at 192.168.203.49@tcp) reconnecting [ 2012.641493] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2012.643496] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2012.649641] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2012.651742] Lustre: Skipped 1 previous similar message [ 2015.703687] Lustre: server umount lustre-OST0000 complete [ 2017.761606] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2017.761994] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2017.762199] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2017.766043] Lustre: Skipped 2 previous similar messages [ 2022.880983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2022.883252] Lustre: Skipped 2 previous similar messages [ 2023.113779] Lustre: server umount lustre-MDT0000 complete [ 2024.418248] LustreError: 146946:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760453771 with bad export cookie 2299861989171474586 [ 2024.420740] LustreError: 166-1: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2024.421669] LustreError: 146946:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2024.570756] Lustre: server umount lustre-MDT0001 complete [ 2029.379939] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 2030.469899] Key type lgssc unregistered [ 2030.606486] LNet: 150681:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2030.610499] LNet: Removed LNI 192.168.203.149@tcp [ 2030.920664] Key type .llcrypt unregistered [ 2030.922236] Key type ._llcrypt unregistered [ 2040.099501] alg: No test for adler32 (adler32-zlib) [ 2040.850333] Key type ._llcrypt registered [ 2040.851437] Key type .llcrypt registered [ 2040.885776] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 2041.171491] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2041.401227] Lustre: Lustre: Build Version: 2.15.7_5_g4a1d30d [ 2041.476817] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 2041.478358] LNet: Accept secure, port 988 [ 2043.056126] Key type lgssc registered [ 2043.382566] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2045.967057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2047.104525] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2047.120507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2048.591488] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 2051.436339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2051.554523] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2053.020294] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 2055.571683] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2057.574926] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2059.811097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2059.921587] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2061.226230] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 2063.014390] Lustre: lustre-OST0000: deleting orphan objects from 0x0:170 to 0x0:193 [ 2063.668954] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2067.109598] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:72 to 0x280000400:161 [ 2082.349883] Lustre: lustre-OST0000: evicting a207d5c3-c6f5-48b1-9a03-d3b3c6877eb3 (at 192.168.203.49@tcp) by administrative request [ 2082.353661] Lustre: Skipped 1 previous similar message [ 2108.386262] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2108.386406] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2108.387363] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2108.389537] Lustre: Skipped 1 previous similar message [ 2112.602133] Lustre: server umount lustre-OST0000 complete [ 2118.112743] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2118.115397] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2118.121401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2118.122975] Lustre: Skipped 1 previous similar message [ 2123.745244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2123.747683] Lustre: Skipped 2 previous similar messages [ 2128.352119] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2128.397307] Lustre: server umount lustre-MDT0000 complete [ 2129.663980] LustreError: 152935:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760453876 with bad export cookie 12355164103769147627 [ 2129.666644] LustreError: 166-1: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2129.667225] LustreError: 152935:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2129.777315] Lustre: server umount lustre-MDT0001 complete [ 2134.351142] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 2135.341899] Key type lgssc unregistered [ 2135.453341] LNet: 156329:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2135.455949] LNet: Removed LNI 192.168.203.149@tcp [ 2135.721604] Key type .llcrypt unregistered [ 2135.722735] Key type ._llcrypt unregistered [ 2161.127843] alg: No test for adler32 (adler32-zlib) [ 2161.879376] Key type ._llcrypt registered [ 2161.880785] Key type .llcrypt registered [ 2161.922437] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 2162.390131] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2162.512173] Lustre: Lustre: Build Version: 2.15.7_5_g4a1d30d [ 2162.587152] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 2162.589346] LNet: Accept secure, port 988 [ 2164.168123] Key type lgssc registered [ 2164.507623] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2167.169888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2168.306443] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2168.321087] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2169.719733] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 2172.455892] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2172.574362] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2173.954580] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 2176.423934] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2178.272062] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2180.409199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2180.529379] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2181.790545] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 2183.590560] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:225 [ 2184.208693] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2187.685623] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:72 to 0x280000400:193 [ 2192.866703] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2192.867887] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2192.871528] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2192.875822] Lustre: Skipped 1 previous similar message [ 2192.879025] Lustre: Skipped 1 previous similar message [ 2197.973689] Lustre: server umount lustre-OST0000 complete [ 2203.105666] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2203.106390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2203.112195] Lustre: Skipped 1 previous similar message [ 2203.116706] Lustre: Skipped 1 previous similar message [ 2205.379154] Lustre: server umount lustre-MDT0000 complete [ 2206.627065] LustreError: 161716:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760453953 with bad export cookie 16358051570074044985 [ 2206.629694] LustreError: 166-1: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2206.629790] LustreError: 161716:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2206.754637] Lustre: server umount lustre-MDT0001 complete [ 2211.453427] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 2212.390491] Key type lgssc unregistered [ 2212.502403] LNet: 162518:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2212.505490] LNet: Removed LNI 192.168.203.149@tcp [ 2212.769465] Key type .llcrypt unregistered [ 2212.771157] Key type ._llcrypt unregistered [ 2221.696896] alg: No test for adler32 (adler32-zlib) [ 2222.448296] Key type ._llcrypt registered [ 2222.449076] Key type .llcrypt registered [ 2222.480496] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 2222.816466] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2222.993035] Lustre: Lustre: Build Version: 2.15.7_5_g4a1d30d [ 2223.058370] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 2223.060108] LNet: Accept secure, port 988 [ 2224.632121] Key type lgssc registered [ 2224.957693] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2227.092706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2229.443386] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 10:59:35 (1760453975) [ 2234.418633] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_hostid [ 2237.363262] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 2240.451741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2242.659840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2244.684610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2246.800395] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2248.945742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2248.971053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2250.045154] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2250.054745] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2250.088078] Lustre: lustre-MDT0000: new disk, initializing [ 2250.110928] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2250.117437] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2251.435729] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 2255.595426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2255.619529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2255.650089] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2255.667581] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 2255.669768] Lustre: Skipped 1 previous similar message [ 2255.707314] Lustre: lustre-MDT0001: new disk, initializing [ 2255.740191] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2255.752979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2255.757454] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2257.083225] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 2260.913347] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2262.760143] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2264.883147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2264.905695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2264.907947] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2264.989485] Lustre: lustre-OST0000: new disk, initializing [ 2264.992212] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2265.018723] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2266.329560] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 2269.973401] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2271.732902] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2271.736765] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2280.929709] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2280.931633] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2280.934583] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2281.953813] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2281.954310] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2281.959246] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2284.247370] Lustre: server umount lustre-OST0000 complete [ 2286.560625] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2286.563389] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2286.567371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2286.568943] Lustre: Skipped 1 previous similar message [ 2291.522782] Lustre: server umount lustre-MDT0000 complete [ 2292.192731] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2292.196476] LustreError: Skipped 1 previous similar message [ 2292.709934] LustreError: 167769:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760454039 with bad export cookie 3213774253574120227 [ 2292.712279] LustreError: 166-1: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2292.713811] LustreError: 167769:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2292.825036] Lustre: server umount lustre-MDT0001 complete [ 2297.404873] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 2298.318536] Key type lgssc unregistered [ 2298.427260] LNet: 170917:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2298.429540] LNet: Removed LNI 192.168.203.149@tcp [ 2298.709248] Key type .llcrypt unregistered [ 2298.710178] Key type ._llcrypt unregistered [ 2307.124870] alg: No test for adler32 (adler32-zlib) [ 2307.876309] Key type ._llcrypt registered [ 2307.877261] Key type .llcrypt registered [ 2307.907663] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 2308.277491] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2308.423360] Lustre: Lustre: Build Version: 2.15.7_5_g4a1d30d [ 2308.494174] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 2308.495531] LNet: Accept secure, port 988 [ 2310.072116] Key type lgssc registered [ 2310.389525] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2312.866319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2313.994481] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2314.009941] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2315.409100] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 2318.090811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2318.188094] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2319.512916] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 2321.863692] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2323.628899] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2325.696662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2325.794053] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2327.100941] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 2328.868924] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 2329.456654] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2336.008805] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2336.014903] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2336.021355] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2336.024688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2336.027909] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.149@tcp (at 0@lo) [ 2336.028899] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:65 [ 2336.344939] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2336.350073] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:97 [ 2336.666673] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2336.669848] Lustre: Skipped 1 previous similar message [ 2336.675569] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2336.680652] Lustre: Skipped 1 previous similar message [ 2336.682609] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2336.684621] Lustre: Skipped 1 previous similar message [ 2336.686328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2336.690619] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.149@tcp (at 0@lo) [ 2336.690784] Lustre: lustre-OST0000: deleting orphan objects from 0x0:165 to 0x0:193 [ 2336.693562] Lustre: Skipped 1 previous similar message [ 2336.985891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2336.989046] Lustre: lustre-OST0000: deleting orphan objects from 0x0:165 to 0x0:225 [ 2337.288540] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2337.292390] Lustre: lustre-OST0000: deleting orphan objects from 0x0:165 to 0x0:257 [ 2337.578476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2337.582090] Lustre: lustre-OST0000: deleting orphan objects from 0x0:165 to 0x0:289 [ 2337.889369] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2337.892527] Lustre: Skipped 3 previous similar messages [ 2337.897772] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2337.902907] Lustre: Skipped 3 previous similar messages [ 2337.904309] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2337.907126] Lustre: Skipped 3 previous similar messages [ 2337.908868] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2337.913619] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.149@tcp (at 0@lo) [ 2337.913878] Lustre: lustre-OST0000: deleting orphan objects from 0x0:165 to 0x0:321 [ 2337.916773] Lustre: Skipped 3 previous similar messages [ 2338.230091] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2338.235029] Lustre: lustre-OST0000: deleting orphan objects from 0x0:165 to 0x0:353 [ 2338.523227] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2338.529484] Lustre: lustre-OST0000: deleting orphan objects from 0x0:165 to 0x0:385 [ 2338.813906] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2338.819070] Lustre: lustre-OST0000: deleting orphan objects from 0x0:165 to 0x0:417 [ 2343.118372] Lustre: server umount lustre-OST0000 complete [ 2348.513342] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2348.513848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2348.516424] Lustre: Skipped 4 previous similar messages [ 2348.520295] Lustre: Skipped 1 previous similar message [ 2349.536159] Lustre: 176038:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760454090/real 1760454090] req@000000009ddb3c7c x1845969871189248/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1760454096 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2349.537740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2349.544370] Lustre: 176038:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2349.547068] Lustre: Skipped 1 previous similar message [ 2351.620457] Lustre: server umount lustre-MDT0000 complete [ 2352.822946] LustreError: 172359:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760454099 with bad export cookie 15642312110774690358 [ 2352.825780] LustreError: 166-1: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2352.827485] LustreError: 172359:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2352.961812] Lustre: server umount lustre-MDT0001 complete [ 2357.411929] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 2358.356950] Key type lgssc unregistered [ 2358.462493] LNet: 176844:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2358.464586] LNet: Removed LNI 192.168.203.149@tcp [ 2358.721455] Key type .llcrypt unregistered [ 2358.722318] Key type ._llcrypt unregistered [ 2367.007447] alg: No test for adler32 (adler32-zlib) [ 2367.759294] Key type ._llcrypt registered [ 2367.760336] Key type .llcrypt registered [ 2367.791444] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 2368.152766] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2368.279277] Lustre: Lustre: Build Version: 2.15.7_5_g4a1d30d [ 2368.343974] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 2368.345605] LNet: Accept secure, port 988 [ 2369.920107] Key type lgssc registered [ 2370.227301] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2372.648027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2373.772925] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2373.786254] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2375.128829] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 2377.789666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2377.891365] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2379.182712] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 2381.530030] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2383.342974] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2385.525642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2385.619789] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2386.943371] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 2388.716499] Lustre: lustre-OST0000: deleting orphan objects from 0x0:165 to 0x0:449 [ 2389.401353] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2395.895810] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2396.471221] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2396.476233] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2396.478974] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2396.483262] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.149@tcp (at 0@lo) [ 2396.483277] LustreError: 178326:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 2398.178651] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2398.178751] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2398.179443] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2398.184355] Lustre: Skipped 1 previous similar message [ 2400.696368] Lustre: lustre-OST0000: Not available for connect from 192.168.203.49@tcp (stopping) [ 2400.698884] Lustre: Skipped 1 previous similar message [ 2403.157342] Lustre: server umount lustre-OST0000 complete [ 2416.560089] LustreError: 178326:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 2107 awake [ 2416.563328] LustreError: 178326:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -11 [ 2430.960946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2431.055290] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2432.310770] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 2434.647141] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2435.147314] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 2437.000171] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9138c8e42000.ost_server_uuid 40 [ 2437.496944] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9138c8e42000.ost_server_uuid in FULL state after 0 sec [ 2450.469224] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2450.472524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2450.472568] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.203.149@tcp (at 0@lo) [ 2450.475474] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:66 to 0x280000400:97 [ 2450.480846] Lustre: lustre-OST0000: deleting orphan objects from 0x0:165 to 0x0:481 [ 2450.483146] Lustre: Skipped 1 previous similar message [ 2455.522507] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2455.523205] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2455.527406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2455.528867] LustreError: Skipped 1 previous similar message [ 2455.532542] Lustre: Skipped 1 previous similar message [ 2457.754692] Lustre: server umount lustre-OST0000 complete [ 2460.128930] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2460.132517] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2460.136308] Lustre: Skipped 1 previous similar message [ 2460.137849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2465.090385] Lustre: server umount lustre-MDT0000 complete [ 2465.761086] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2465.765261] LustreError: Skipped 1 previous similar message [ 2466.312206] LustreError: 178287:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760454212 with bad export cookie 4001481104413510199 [ 2466.316080] LustreError: 166-1: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2466.317579] LustreError: 178287:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2471.050119] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 2472.057102] Key type lgssc unregistered [ 2472.181380] LNet: 183619:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2472.184730] LNet: Removed LNI 192.168.203.149@tcp [ 2472.482549] Key type .llcrypt unregistered [ 2472.483937] Key type ._llcrypt unregistered [ 2483.637470] alg: No test for adler32 (adler32-zlib) [ 2484.388280] Key type ._llcrypt registered [ 2484.389624] Key type .llcrypt registered [ 2484.432476] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 2484.816230] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2485.103473] Lustre: Lustre: Build Version: 2.15.7_5_g4a1d30d [ 2485.190706] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 2485.193187] LNet: Accept secure, port 988 [ 2486.768233] Key type lgssc registered [ 2487.150941] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2490.311180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2491.440693] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2491.452603] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2492.935814] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 2495.592233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2495.694138] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2497.159518] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 2499.862226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2499.981275] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2501.341827] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 2504.030666] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2504.057274] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2504.061293] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 2504.114920] Lustre: lustre-OST0001: new disk, initializing [ 2504.116633] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 2504.146319] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2505.531896] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 2506.489820] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2506.493532] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2507.565245] Lustre: lustre-OST0000: deleting orphan objects from 0x0:483 to 0x0:513 [ 2510.837159] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2516.411801] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 2531.809275] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2531.811668] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2531.815618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2533.345547] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2533.346938] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2533.348073] Lustre: Skipped 2 previous similar messages [ 2533.354545] Lustre: Skipped 2 previous similar messages [ 2534.597423] Lustre: server umount lustre-MDT0000 complete [ 2535.757267] LustreError: 185376:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760454282 with bad export cookie 15878802338345872352 [ 2535.759797] LustreError: 166-1: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2535.761959] LustreError: 185376:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2535.883247] Lustre: server umount lustre-MDT0001 complete [ 2542.560189] Lustre: 189178:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760454283/real 1760454283] req@00000000226d6c37 x1845970055767552/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1760454289 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 2542.588244] Lustre: server umount lustre-OST0000 complete [ 2550.240131] Lustre: 189379:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760454290/real 1760454290] req@00000000b87fbdb2 x1845970055768064/t0(0) o39->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1760454296 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 2550.296008] Lustre: server umount lustre-OST0001 complete [ 2551.274127] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 2551.305109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2551.637132] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 2551.665849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2551.970648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 2551.987288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2552.290964] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 2552.307928] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2556.924506] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 2560.279150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2560.421732] LustreError: 137-5: mylustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2560.442443] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2561.919666] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 2564.698310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2566.311459] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 2569.193175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2569.293920] Lustre: mylustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2569.297615] Lustre: Skipped 1 previous similar message [ 2570.340141] LustreError: 137-5: mylustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2570.686368] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 2573.370199] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2574.767524] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 2576.428385] Lustre: mylustre-OST0001: deleting orphan objects from 0x0:3 to 0x0:33 [ 2576.428844] Lustre: mylustre-OST0000: deleting orphan objects from 0x0:515 to 0x0:545 [ 2583.370157] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2584.619027] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 2595.809631] LustreError: 11-0: mylustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2595.813344] Lustre: mylustre-MDT0000-osp-MDT0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2595.819429] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2595.821984] Lustre: Skipped 1 previous similar message [ 2595.912755] Lustre: server umount mylustre-MDT0000 complete [ 2596.834145] LustreError: 137-5: mylustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2596.840442] LustreError: Skipped 3 previous similar messages [ 2597.146467] LustreError: 190819:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760454343 with bad export cookie 15878802338345875033 [ 2597.149345] LustreError: 166-1: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2597.151918] LustreError: 190819:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2607.072147] Lustre: 194540:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760454347/real 1760454347] req@00000000b95d316d x1845970055791936/t0(0) o39->mylustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1760454353 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 2607.097385] Lustre: server umount mylustre-OST0000 complete [ 2607.099477] Lustre: Skipped 1 previous similar message [ 2613.728154] Lustre: 194741:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760454354/real 1760454354] req@00000000d95bad87 x1845970055792384/t0(0) o39->mylustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1760454360 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 2614.869259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 2614.892297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2615.243709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 2615.273542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2615.625122] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 2615.650540] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2616.007519] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 2616.041484] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2620.681571] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 2623.967163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2624.152995] LustreError: 137-5: tfs-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2624.176828] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2624.178730] Lustre: Skipped 1 previous similar message [ 2625.610500] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 2628.492710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2630.117390] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 2633.029402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2634.148747] LustreError: 137-5: tfs-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2634.640568] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 2637.591990] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2639.022421] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 2639.785100] Lustre: tfs-OST0001: deleting orphan objects from 0x0:35 to 0x0:65 [ 2639.785740] Lustre: tfs-OST0000: deleting orphan objects from 0x0:515 to 0x0:577 [ 2642.202113] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2643.436409] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 2654.177122] LustreError: 11-0: tfs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2654.180046] Lustre: tfs-MDT0000-osp-MDT0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2654.182862] Lustre: Skipped 3 previous similar messages [ 2654.184294] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 2656.519879] Lustre: server umount tfs-MDT0000 complete [ 2656.521623] Lustre: Skipped 1 previous similar message [ 2657.675970] LustreError: 196179:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760454404 with bad export cookie 15878802338345876867 [ 2657.678152] LustreError: 166-1: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2657.680304] LustreError: 196179:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2664.416133] Lustre: 199839:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760454405/real 1760454405] req@00000000d16e306f x1845970055811456/t0(0) o39->tfs-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1760454411 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 2673.182558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 2673.211565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2673.570728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 2673.598267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2673.949837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 2673.973858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2674.301233] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 2674.327968] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2678.544530] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 2681.648924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2681.791312] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2681.814646] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2681.816495] Lustre: Skipped 3 previous similar messages [ 2683.221537] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 2685.954437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2687.435096] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 2690.060120] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2691.478269] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 2694.157087] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2695.521824] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 2696.871389] Lustre: lustre-OST0001: deleting orphan objects from 0x0:35 to 0x0:97 [ 2696.871418] Lustre: lustre-OST0000: deleting orphan objects from 0x0:515 to 0x0:609 [ 2703.541408] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2704.700135] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 2711.952176] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 11:07:38 (1760454458) [ 2726.880978] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2726.883027] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2726.885816] Lustre: Skipped 3 previous similar messages [ 2726.887123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2726.888611] Lustre: Skipped 4 previous similar messages [ 2729.740035] Lustre: server umount lustre-MDT0000 complete [ 2729.741548] Lustre: Skipped 3 previous similar messages [ 2730.904881] LustreError: 202302:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760454477 with bad export cookie 15878802338345878498 [ 2730.907650] LustreError: 166-1: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2730.907802] LustreError: 202302:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2733.025193] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2733.028336] LustreError: Skipped 2 previous similar messages [ 2735.073386] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2735.075006] Lustre: Skipped 7 previous similar messages [ 2745.312104] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2768.352118] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2770.514612] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_hostid [ 2773.353563] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 2776.665750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2778.721461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2780.659199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2782.693184] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2784.827078] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2784.847532] LDISKFS-fs (dm-0): Mount option "noacl" will be removed by 3.5 [ 2784.847532] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 2784.847532] [ 2784.854777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 2784.935929] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2784.945300] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2784.980257] Lustre: lustre-MDT0000: new disk, initializing [ 2785.003193] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2785.005959] Lustre: Skipped 3 previous similar messages [ 2785.011582] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2786.411980] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 2790.328711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2790.349089] LDISKFS-fs (dm-1): Mount option "noacl" will be removed by 3.5 [ 2790.349089] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 2790.349089] [ 2790.355193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 2790.397956] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 2790.400082] Lustre: Skipped 1 previous similar message [ 2790.434218] Lustre: lustre-MDT0001: new disk, initializing [ 2790.470470] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2790.474108] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2791.887813] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 2796.014025] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2796.037971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2796.040720] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2796.124730] Lustre: lustre-OST0000: new disk, initializing [ 2796.126350] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2797.520545] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 2797.559969] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2797.563746] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2801.545100] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2801.571425] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2801.574893] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 2801.622262] Lustre: lustre-OST0001: new disk, initializing [ 2801.623770] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 2802.685180] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2802.688602] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2803.029835] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 2807.217117] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 11:09:13 (1760454553) [ 2810.650594] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 11:09:16 (1760454556) [ 2823.138590] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2823.139075] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2823.141506] LustreError: Skipped 1 previous similar message [ 2823.146313] Lustre: Skipped 5 previous similar messages [ 2823.146900] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2823.151166] Lustre: Skipped 5 previous similar messages [ 2828.013840] Lustre: server umount lustre-OST0000 complete [ 2828.015032] Lustre: Skipped 3 previous similar messages [ 2828.259380] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2828.263114] LustreError: Skipped 7 previous similar messages [ 2831.328650] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2836.599892] LustreError: 207765:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760454583 with bad export cookie 15878802338345880017 [ 2836.602495] LustreError: 166-1: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2836.604084] LustreError: 207765:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2847.131937] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 2858.727204] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_hostid [ 2864.459995] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 2865.220584] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2867.740207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2869.904307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2871.867221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2873.725500] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2875.832784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2875.854853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2875.927642] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2875.929595] Lustre: Skipped 1 previous similar message [ 2875.936946] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2875.970402] Lustre: lustre-MDT0000: new disk, initializing [ 2875.994982] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2877.296402] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 2881.246085] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2881.268459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2881.395159] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2882.779464] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 2886.263016] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2887.948290] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2889.926335] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2889.949169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2889.951878] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2891.322899] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2891.489661] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 2895.316538] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2901.474577] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2901.474782] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2901.477527] Lustre: Skipped 5 previous similar messages [ 2901.482879] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2901.484589] Lustre: Skipped 8 previous similar messages [ 2919.392119] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2920.677151] LustreError: 214945:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760454667 with bad export cookie 15878802338345881445 [ 2920.680035] LustreError: 166-1: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2920.680079] LustreError: 214945:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2925.438290] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 2926.465488] Key type lgssc unregistered [ 2926.592302] LNet: 218803:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2926.594709] LNet: Removed LNI 192.168.203.149@tcp [ 2926.871941] Key type .llcrypt unregistered [ 2926.872928] Key type ._llcrypt unregistered [ 2936.257787] alg: No test for adler32 (adler32-zlib) [ 2937.009405] Key type ._llcrypt registered [ 2937.011010] Key type .llcrypt registered [ 2937.050163] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 2937.472628] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2937.632246] Lustre: Lustre: Build Version: 2.15.7_5_g4a1d30d [ 2937.703416] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 2937.704982] LNet: Accept secure, port 988 [ 2939.280111] Key type lgssc registered [ 2939.659693] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2942.334491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2943.476400] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2943.491455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2944.995845] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 2947.835917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2947.967773] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2949.417837] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 2951.957683] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2953.843910] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2956.020157] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2956.132668] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2957.705340] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 2959.206833] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 2960.201976] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2960.542715] Lustre: Setting parameter lustre-OST0000.ost.unknown_param in log lustre-OST0000 [ 2963.298925] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2963.300346] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2965.486642] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2965.489734] Lustre: Skipped 1 previous similar message [ 2967.333400] Lustre: server umount lustre-OST0000 complete [ 2973.666425] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2973.666482] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2973.667101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2973.674577] Lustre: Skipped 1 previous similar message [ 2974.728098] Lustre: server umount lustre-MDT0000 complete [ 2975.968307] LustreError: 220294:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760454722 with bad export cookie 4419440091601577548 [ 2975.971660] LustreError: 220294:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2975.971946] LustreError: 166-1: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2976.100570] Lustre: server umount lustre-MDT0001 complete [ 2979.893237] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 2980.971759] Key type lgssc unregistered [ 2981.083381] LNet: 224017:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2981.085799] LNet: Removed LNI 192.168.203.149@tcp [ 2981.379437] Key type .llcrypt unregistered [ 2981.380464] Key type ._llcrypt unregistered [ 2987.945591] alg: No test for adler32 (adler32-zlib) [ 2988.697410] Key type ._llcrypt registered [ 2988.698975] Key type .llcrypt registered [ 2988.742551] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 2989.299037] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2989.526215] Lustre: Lustre: Build Version: 2.15.7_5_g4a1d30d [ 2989.612436] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 2989.614911] LNet: Accept secure, port 988 [ 2991.200105] Key type lgssc registered [ 2991.554499] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2994.623168] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3011.104269] LustreError: 225259:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.203.149@tcp: failed processing log, type 1: rc = -5 [ 3042.848283] LustreError: 225259:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.203.149@tcp: failed processing log, type 4: rc = -110 [ 3072.544286] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3073.949029] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 3076.321802] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3077.092803] Lustre: server umount lustre-OST0000 complete [ 3082.497278] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 3083.627656] Key type lgssc unregistered [ 3083.749374] LNet: 226901:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3083.753372] LNet: Removed LNI 192.168.203.149@tcp [ 3084.062207] Key type .llcrypt unregistered [ 3084.063953] Key type ._llcrypt unregistered [ 3097.145969] alg: No test for adler32 (adler32-zlib) [ 3097.897415] Key type ._llcrypt registered [ 3097.898460] Key type .llcrypt registered [ 3097.942964] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 3098.474649] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3098.671497] Lustre: Lustre: Build Version: 2.15.7_5_g4a1d30d [ 3098.749350] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 3098.750858] LNet: Accept secure, port 988 [ 3100.328149] Key type lgssc registered [ 3100.683700] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3105.006190] loop0: detected capacity change from 0 to 204800000 [ 3105.045174] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3105.049096] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3105.175598] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3105.179290] blk_update_request: operation not supported error, dev loop0, sector 160240 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3105.202462] blk_update_request: operation not supported error, dev loop0, sector 160344 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3105.221829] blk_update_request: operation not supported error, dev loop0, sector 168552 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3105.237235] blk_update_request: operation not supported error, dev loop0, sector 173280 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3105.261393] blk_update_request: operation not supported error, dev loop0, sector 176912 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3105.289910] blk_update_request: operation not supported error, dev loop0, sector 185112 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3105.318805] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3105.640997] loop0: detected capacity change from 0 to 204800000 [ 3105.687122] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3105.928365] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3106.262478] loop0: detected capacity change from 0 to 204800000 [ 3106.418875] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3106.744647] loop0: detected capacity change from 0 to 204800000 [ 3106.893863] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3107.204110] loop0: detected capacity change from 0 to 204800000 [ 3107.217769] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3108.531917] loop0: detected capacity change from 0 to 204800000 [ 3108.541867] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3109.852242] loop0: detected capacity change from 0 to 204800000 [ 3109.864073] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3111.163609] loop0: detected capacity change from 0 to 204800000 [ 3111.174407] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3112.449047] loop0: detected capacity change from 0 to 204800000 [ 3112.478562] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3112.500526] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3112.528529] Lustre: 230355:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 3112.531802] Lustre: 230355:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 3112.535141] Lustre: 230355:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 3112.538899] Lustre: 230355:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 3112.542547] Lustre: 230355:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 3112.545845] Lustre: 230355:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 3112.549376] Lustre: 230355:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 3112.552846] Lustre: 230355:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 3112.555961] Lustre: 230355:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 3112.559263] Lustre: 230355:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 3112.562282] Lustre: 230355:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 3112.565242] Lustre: 230355:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 3112.568641] Lustre: 230355:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 3112.572215] Lustre: 230355:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 3121.120099] Lustre: 230634:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760454861/real 1760454861] req@000000004c5bb6f2 x1845970699550848/t0(0) o251->MGC192.168.203.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1760454867 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3121.130300] Lustre: server umount MGS complete [ 3121.443042] loop0: detected capacity change from 0 to 204800000 [ 3121.476779] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3121.620140] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3121.634434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3121.648961] LustreError: 230688:0:(mdt_handler.c:7428:mdt_iocontrol()) lustre-MDT0000: Aborting recovery for device [ 3121.953353] loop1: detected capacity change from 0 to 204800000 [ 3121.985680] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3122.029548] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3122.053400] Lustre: 230804:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 3122.057608] Lustre: 230804:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 3122.060933] Lustre: 230804:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 3122.064519] Lustre: 230804:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 3122.068961] Lustre: 230804:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 3122.073890] Lustre: 230804:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 3122.078044] Lustre: 230804:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 3122.084489] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3122.159956] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3122.194414] LustreError: 230804:0:(mdt_handler.c:7428:mdt_iocontrol()) lustre-MDT0001: Aborting recovery for device [ 3122.496865] loop2: detected capacity change from 0 to 204800000 [ 3122.533160] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3122.570757] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3122.575127] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/loop2 [ 3122.597934] Lustre: 230887:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 3122.603585] Lustre: 230887:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 3122.609201] Lustre: 230887:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 3122.613372] Lustre: 230887:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 3122.617164] Lustre: 230887:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 3122.620276] Lustre: 230887:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 3122.624444] Lustre: 230887:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 3122.677400] Lustre: Found index 0 for lustre-OST0000, updating log [ 3122.711183] LustreError: 230887:0:(ofd_obd.c:1263:ofd_iocontrol()) lustre-OST0000: aborting recovery [ 3123.033234] loop3: detected capacity change from 0 to 204800000 [ 3123.075103] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3123.111783] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3123.114701] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/loop3 [ 3123.141151] Lustre: 230974:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 3123.147205] Lustre: 230974:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 3123.153649] Lustre: 230974:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 3123.156693] Lustre: 230974:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 3123.160280] Lustre: 230974:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 3123.166951] Lustre: 230974:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 3123.170457] Lustre: 230974:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 3123.210719] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3123.214046] Lustre: Skipped 1 previous similar message [ 3123.215346] LustreError: 230747:0:(lod_dev.c:424:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000 get update log failed: rc = -108 [ 3123.221956] LustreError: 230974:0:(ofd_obd.c:1263:ofd_iocontrol()) lustre-OST0001: aborting recovery [ 3127.267564] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:5 to 0x280000400:33 [ 3127.267990] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:5 to 0x2c0000400:33 [ 3127.268114] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6 to 0x0:33 [ 3127.270921] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6 to 0x0:33 [ 3128.505864] LustreError: 231213:0:(mdt_identity.c:112:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 276us: rc = -2 [ 3142.626342] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3142.626438] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3142.626828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3142.629959] Lustre: Skipped 3 previous similar messages [ 3145.792918] Lustre: server umount lustre-MDT0000 complete [ 3146.118679] LustreError: 230695:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760454892 with bad export cookie 1859932823424987937 [ 3146.120287] LustreError: 166-1: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3146.121739] LustreError: 230695:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3147.744885] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3147.745210] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3147.745512] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3147.745518] Lustre: Skipped 3 previous similar messages [ 3147.748615] LustreError: Skipped 1 previous similar message [ 3147.755424] Lustre: Skipped 1 previous similar message [ 3151.776754] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3151.776775] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3151.778850] Lustre: Skipped 2 previous similar messages [ 3151.782091] LustreError: Skipped 1 previous similar message [ 3152.321141] Lustre: server umount lustre-MDT0001 complete [ 3158.764164] Lustre: server umount lustre-OST0000 complete [ 3173.344120] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3173.410640] Lustre: server umount lustre-OST0001 complete [ 3176.021285] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 11:15:22 (1760454922) [ 3185.099977] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_hostid [ 3188.167262] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 3191.608195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3193.761847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3195.806820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3197.903920] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3200.261152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3200.293820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3200.381290] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3200.391141] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3200.427439] Lustre: lustre-MDT0000: new disk, initializing [ 3200.453758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3200.461523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3201.885197] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 3206.014176] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3206.036083] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3206.067422] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3206.083715] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3206.087235] Lustre: Skipped 1 previous similar message [ 3206.121571] Lustre: lustre-MDT0001: new disk, initializing [ 3206.148099] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3206.158403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3206.160943] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3207.628788] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 3211.478826] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3213.547981] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3215.810476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3215.834113] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3215.838353] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3215.930544] Lustre: lustre-OST0000: new disk, initializing [ 3215.932268] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3215.960298] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3217.373462] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 3217.526494] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3217.530839] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3221.107987] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3223.421616] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 3228.522462] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 3236.893462] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb in log lustre-client [ 3279.333441] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3279.334143] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3279.338777] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3279.340076] LustreError: Skipped 1 previous similar message [ 3279.346520] Lustre: Skipped 1 previous similar message [ 3281.447990] Lustre: server umount lustre-OST0000 complete [ 3282.912895] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3282.916552] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3282.922078] Lustre: Skipped 1 previous similar message [ 3284.449462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3284.451353] Lustre: Skipped 2 previous similar messages [ 3289.569265] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3289.574049] LustreError: Skipped 1 previous similar message [ 3290.177799] LustreError: 235993:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760455036 with bad export cookie 1859932823424990604 [ 3290.183328] LustreError: 166-1: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3290.185572] LustreError: 235993:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3293.579034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3295.062869] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 3303.392196] Lustre: 240674:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760455043/real 1760455043] req@00000000e1c3cee4 x1845970699611008/t0(0) o251->MGC192.168.203.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1760455049 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 3303.418323] Lustre: server umount MGS complete [ 3303.421310] Lustre: Skipped 2 previous similar messages [ 3308.170729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3308.377897] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3308.384400] LustreError: 241309:0:(qmt_pool.c:1415:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 3308.407133] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3310.057461] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 3313.674622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3315.852252] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 3319.351759] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3321.885834] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3325.225693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3327.027291] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 3329.606124] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3367.701086] Lustre: server umount lustre-OST0000 complete [ 3369.441394] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3369.442244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3369.446163] Lustre: Skipped 2 previous similar messages [ 3369.450191] Lustre: Skipped 1 previous similar message [ 3374.560150] Lustre: 244625:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760455115/real 1760455115] req@0000000076d7c31f x1845970699631424/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1760455121 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 3376.608105] Lustre: 228324:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760455116/real 1760455116] req@00000000f40dcaf4 x1845970699631488/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1760455123 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'osp-pre-0-1.0' [ 3376.617842] Lustre: 228324:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3378.377647] LustreError: 241277:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760455124 with bad export cookie 1859932823424992389 [ 3378.383893] LustreError: 166-1: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3378.385450] LustreError: 241277:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3383.466554] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 3384.669829] Key type lgssc unregistered [ 3384.803490] LNet: 245429:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3384.809212] LNet: Removed LNI 192.168.203.149@tcp [ 3385.256767] Key type .llcrypt unregistered [ 3385.258706] Key type ._llcrypt unregistered [ 3402.294105] alg: No test for adler32 (adler32-zlib) [ 3403.047426] Key type ._llcrypt registered [ 3403.048316] Key type .llcrypt registered [ 3403.105530] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_hostid [ 3409.316250] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 3409.625383] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3409.854327] Lustre: Lustre: Build Version: 2.15.7_5_g4a1d30d [ 3409.926087] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 3409.927820] LNet: Accept secure, port 988 [ 3411.512158] Key type lgssc registered [ 3412.098150] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3416.230604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3419.385399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3421.393709] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3423.398230] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3425.599823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3425.623911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3426.697840] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3426.706662] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3426.742688] Lustre: lustre-MDT0000: new disk, initializing [ 3426.764796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3426.770655] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3428.064672] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 3432.327779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3432.351722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3432.382874] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3432.399125] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3432.400741] Lustre: Skipped 1 previous similar message [ 3432.439269] Lustre: lustre-MDT0001: new disk, initializing [ 3432.470448] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3432.482883] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3432.486533] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3433.990316] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 3438.419193] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3440.710952] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3443.264544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3443.294961] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3443.297057] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3443.395590] Lustre: lustre-OST0000: new disk, initializing [ 3443.397568] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3443.430344] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3445.468783] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 3450.034159] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3452.927865] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3452.932632] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3458.636994] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 3462.537575] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 3469.777084] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb in log lustre-client [ 3524.579403] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3524.579600] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3524.579909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3524.582806] Lustre: Skipped 1 previous similar message [ 3524.588482] LustreError: Skipped 1 previous similar message [ 3527.545130] Lustre: server umount lustre-OST0000 complete [ 3529.698218] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3529.698753] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3529.699245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3529.699254] Lustre: Skipped 1 previous similar message [ 3529.705713] Lustre: Skipped 1 previous similar message [ 3534.817269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3534.818973] Lustre: Skipped 2 previous similar messages [ 3535.052844] Lustre: server umount lustre-MDT0000 complete [ 3536.234201] LustreError: 249496:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760455282 with bad export cookie 9419506869505096861 [ 3536.237622] LustreError: 166-1: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3536.239389] LustreError: 249496:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3536.385609] Lustre: server umount lustre-MDT0001 complete [ 3539.222043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3540.620803] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 3547.616145] Lustre: 254937:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760455288/real 1760455288] req@000000000f7d4dbf x1845971025702016/t0(0) o251->MGC192.168.203.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1760455294 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3547.632157] Lustre: server umount MGS complete [ 3551.384878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3551.554530] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3551.561200] LustreError: 255572:0:(qmt_pool.c:1415:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 3551.587218] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3553.117161] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 3555.856705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3555.971366] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3557.314753] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 3559.723705] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3561.551164] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3563.676755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3565.135089] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 3567.773131] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3587.553085] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3587.555871] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3587.559512] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3591.654294] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3591.654938] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3591.661662] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3593.644242] Lustre: server umount lustre-OST0000 complete [ 3596.769824] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3596.774549] Lustre: Skipped 1 previous similar message [ 3601.889643] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3601.897558] LustreError: Skipped 1 previous similar message [ 3602.788188] LustreError: 255539:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760455349 with bad export cookie 9419506869505098660 [ 3602.791736] LustreError: 166-1: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3602.793062] LustreError: 255539:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3602.924852] Lustre: server umount lustre-MDT0001 complete [ 3602.926393] Lustre: Skipped 1 previous similar message [ 3608.112735] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 3609.152585] Key type lgssc unregistered [ 3609.268405] LNet: 259669:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3609.270847] LNet: Removed LNI 192.168.203.149@tcp [ 3609.561100] Key type .llcrypt unregistered [ 3609.562527] Key type ._llcrypt unregistered [ 3621.387855] alg: No test for adler32 (adler32-zlib) [ 3622.141337] Key type ._llcrypt registered [ 3622.142543] Key type .llcrypt registered [ 3622.186279] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 3622.539983] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3622.826087] Lustre: Lustre: Build Version: 2.15.7_5_g4a1d30d [ 3622.896782] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 3622.898232] LNet: Accept secure, port 988 [ 3624.472087] Key type lgssc registered [ 3624.924149] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3628.511376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3629.673367] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3629.690883] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3631.671526] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 3635.170952] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3635.445566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3635.596307] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3637.666750] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 3640.803469] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3642.714374] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3645.076593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3645.225350] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3646.861625] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 3649.477933] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3651.835620] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3651.863128] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3651.865432] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 3651.920988] Lustre: lustre-OST0001: new disk, initializing [ 3651.922758] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 3651.952413] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3653.475151] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 3658.776676] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3658.784269] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3658.812678] LustreError: 264664:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 3659.878947] LustreError: 264664:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 3659.885157] LustreError: 264664:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [ 3660.902305] LustreError: 264671:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 3660.911508] LustreError: 264671:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [ 3662.948219] LustreError: 264671:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 3662.953989] LustreError: 264671:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) Skipped 3 previous similar messages [ 3663.739757] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3663.873057] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3665.899410] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9138c2e3f800.ost_server_uuid 40 [ 3666.422249] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9138c2e3f800.ost_server_uuid in FULL state after 0 sec [ 3667.045644] LustreError: 264671:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 3667.047992] LustreError: 264671:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) Skipped 7 previous similar messages [ 3668.280468] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9138c2e3f800.ost_server_uuid 40 [ 3668.709686] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9138c2e3f800.ost_server_uuid in FULL state after 0 sec [ 3680.228024] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3680.231584] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3680.235072] Lustre: Skipped 1 previous similar message [ 3682.232716] Lustre: lustre-OST0001: Not available for connect from 192.168.203.49@tcp (stopping) [ 3685.348481] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3685.349965] Lustre: Skipped 1 previous similar message [ 3686.060261] Lustre: server umount lustre-OST0001 complete [ 3690.466591] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3690.466944] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3690.467778] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3690.467850] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3690.470233] LustreError: Skipped 1 previous similar message [ 3690.475443] Lustre: Skipped 2 previous similar messages [ 3693.734645] Lustre: server umount lustre-OST0000 complete [ 3695.585509] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3695.586350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3695.590281] Lustre: Skipped 1 previous similar message [ 3695.592479] Lustre: Skipped 1 previous similar message [ 3701.136519] Lustre: server umount lustre-MDT0000 complete [ 3702.416077] LustreError: 262017:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760455449 with bad export cookie 1546078386336578767 [ 3702.420582] LustreError: 166-1: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3707.025120] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 3708.102872] Key type lgssc unregistered [ 3708.216505] LNet: 266865:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3708.219085] LNet: Removed LNI 192.168.203.149@tcp [ 3708.506261] Key type .llcrypt unregistered [ 3708.507249] Key type ._llcrypt unregistered [ 3721.773247] alg: No test for adler32 (adler32-zlib) [ 3722.526690] Key type ._llcrypt registered [ 3722.527711] Key type .llcrypt registered [ 3722.568150] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 3722.854621] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3723.126754] Lustre: Lustre: Build Version: 2.15.7_5_g4a1d30d [ 3723.206120] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 3723.208884] LNet: Accept secure, port 988 [ 3724.784083] Key type lgssc registered [ 3725.108901] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3727.764684] loop0: detected capacity change from 0 to 1099511627776 [ 3727.902282] SGI XFS with ACLs, security attributes, quota, no debug enabled [ 3727.904840] *** VALIDATE xfs *** [ 3727.910839] XFS (loop0): Mounting V5 Filesystem [ 3727.919555] XFS (loop0): Ending clean mount [ 3728.334062] loop1: detected capacity change from 0 to 18691697672192 [ 3744.920588] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3752.394113] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_hostid [ 3755.663417] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 3760.123381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3763.159740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3766.019480] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3768.971240] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3772.200466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3772.229415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3773.317469] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3773.326205] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3773.362475] Lustre: lustre-MDT0000: new disk, initializing [ 3773.386239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3773.392783] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3775.188035] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 3780.626287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3780.657352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3780.708643] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3780.732613] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3780.734739] Lustre: Skipped 1 previous similar message [ 3780.777104] Lustre: lustre-MDT0001: new disk, initializing [ 3780.813483] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3780.829203] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3780.832777] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3782.886177] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 3787.501308] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3789.732409] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3792.345692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3792.369170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3792.371269] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3792.451783] Lustre: lustre-OST0000: new disk, initializing [ 3792.453327] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3792.475492] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3794.032788] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 3798.016147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3798.020150] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3798.118818] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3799.667677] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3799.752467] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3801.177591] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3801.256831] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3803.106661] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3803.106752] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3803.109261] LustreError: Skipped 1 previous similar message [ 3803.109705] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3803.112421] Lustre: Skipped 1 previous similar message [ 3803.118996] Lustre: Skipped 1 previous similar message [ 3808.045504] Lustre: server umount lustre-OST0000 complete [ 3811.808853] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3811.811633] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3811.815856] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3813.345032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3813.345431] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3815.427756] Lustre: server umount lustre-MDT0000 complete [ 3816.661108] LustreError: 271798:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760455563 with bad export cookie 14364772281811258536 [ 3816.664226] LustreError: 166-1: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3816.665056] LustreError: 271798:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3816.784327] Lustre: server umount lustre-MDT0001 complete [ 3819.561813] XFS (loop0): Unmounting Filesystem [ 3821.168977] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 11:26:07 (1760455567) [ 3823.303205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3823.453935] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3823.472365] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3824.778416] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 3827.443635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3827.552939] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3829.057664] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 3831.500455] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3833.352659] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3835.512677] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3835.617843] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3836.952498] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 3839.442135] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3848.162495] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3848.162645] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3848.163493] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3848.163497] Lustre: Skipped 1 previous similar message [ 3848.170127] Lustre: Skipped 1 previous similar message [ 3853.282582] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3853.285475] Lustre: Skipped 2 previous similar messages [ 3853.650459] Lustre: server umount lustre-OST0000 complete [ 3858.401671] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3858.401833] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3858.407171] Lustre: Skipped 1 previous similar message [ 3863.521194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3863.523356] Lustre: Skipped 3 previous similar messages [ 3869.152090] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3869.192761] Lustre: server umount lustre-MDT0000 complete [ 3870.478900] LustreError: 278447:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760455617 with bad export cookie 14364772281811259390 [ 3870.481731] LustreError: 166-1: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3870.483391] LustreError: 278447:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3875.883632] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 3876.844599] Key type lgssc unregistered [ 3876.966318] LNet: 280248:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3876.970402] LNet: Removed LNI 192.168.203.149@tcp [ 3877.291895] Key type .llcrypt unregistered [ 3877.292886] Key type ._llcrypt unregistered [ 3887.254292] alg: No test for adler32 (adler32-zlib) [ 3888.005386] Key type ._llcrypt registered [ 3888.006302] Key type .llcrypt registered [ 3888.040045] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 3888.406174] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3888.607053] Lustre: Lustre: Build Version: 2.15.7_5_g4a1d30d [ 3888.675105] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 3888.676594] LNet: Accept secure, port 988 [ 3890.256083] Key type lgssc registered [ 3890.589771] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3893.132165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3894.271753] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3894.285227] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3895.890499] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 3899.040021] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3899.149455] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3900.575376] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 3903.432134] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3905.517259] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3907.923558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3908.081328] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3909.610629] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 3912.168255] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3920.864891] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 3920.866876] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3920.869969] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3923.895370] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.49@tcp (stopping) [ 3924.450828] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3926.282973] Lustre: server umount lustre-MDT0001 complete [ 3927.374099] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3929.008917] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 192.168.203.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3930.127051] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3930.171773] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3930.174598] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3930.192617] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3930.327349] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3931.810404] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 3935.716375] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 3935.724208] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to (at 0@lo) [ 3935.725728] Lustre: Skipped 1 previous similar message [ 3939.301309] LustreError: 285806:0:(genops.c:522:class_register_device()) lustre-MDT0001-lwp-OST0000: already exists, won't add [ 3939.303778] LustreError: 285806:0:(obd_mount.c:195:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 3939.305908] LustreError: 285806:0:(obd_mount_server.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 3939.308332] LustreError: 281853:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3939.323953] LustreError: 285809:0:(obd_config.c:1999:class_config_llog_handler()) MGC192.168.203.149@tcp: cfg command failed: rc = -17 [ 3939.327071] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3939.327071] [ 4236.218767] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.49@tcp (stopping) [ 4236.222174] Lustre: Skipped 2 previous similar messages [ 4237.794789] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 4237.796643] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4241.287591] Lustre: server umount lustre-MDT0001 complete [ 4241.329674] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 192.168.203.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4241.333265] LustreError: Skipped 2 previous similar messages [ 4242.206658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4244.690833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4244.722879] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4244.724732] Lustre: Found index 1 for lustre-MDT0001, updating log [ 4244.738125] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4244.842850] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4246.313486] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 4250.084402] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 4250.091135] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 4250.094017] Lustre: Skipped 1 previous similar message [ 4254.695769] LustreError: 298391:0:(genops.c:522:class_register_device()) lustre-MDT0001-lwp-OST0000: already exists, won't add [ 4254.698650] LustreError: 298391:0:(genops.c:522:class_register_device()) Skipped 1 previous similar message [ 4254.701498] LustreError: 298391:0:(obd_mount.c:195:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 4254.706174] LustreError: 298391:0:(obd_mount_server.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 4254.710684] LustreError: 281853:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 4254.714108] LustreError: 281853:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [ 4254.732995] LustreError: 298394:0:(obd_config.c:680:class_setup()) Device 8 already setup (type osp) [ 4254.734856] LustreError: 298394:0:(obd_config.c:1999:class_config_llog_handler()) MGC192.168.203.149@tcp: cfg command failed: rc = -17 [ 4254.737308] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.203.149@tcp [ 4254.737308] [ 4550.116222] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4550.116788] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4550.119341] Lustre: Skipped 2 previous similar messages [ 4550.122184] Lustre: Skipped 3 previous similar messages [ 4555.200590] Lustre: server umount lustre-MDT0001 complete [ 4555.234477] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4555.238867] LustreError: Skipped 3 previous similar messages [ 4556.161783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4558.716698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4558.745775] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4558.748988] Lustre: Found index 1 for lustre-MDT0001, updating log [ 4558.767092] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4558.883055] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 4560.449635] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 4563.939875] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 4563.945926] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 4563.947498] Lustre: Skipped 1 previous similar message [ 4565.990063] LustreError: 310723:0:(genops.c:522:class_register_device()) lustre-MDT0001-lwp-OST0000: already exists, won't add [ 4565.992921] LustreError: 310723:0:(obd_mount.c:195:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 4565.995148] LustreError: 310723:0:(obd_mount_server.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 4566.000535] LustreError: 281853:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 4566.003812] LustreError: 281853:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [ 4566.020550] LustreError: 310726:0:(lod_lov.c:228:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [ 4566.025502] LustreError: 310726:0:(obd_config.c:1999:class_config_llog_handler()) MGC192.168.203.149@tcp: cfg command failed: rc = -114 [ 4566.028519] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [ 4566.028519] [ 4876.259948] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4876.261706] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4876.263425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4876.263441] Lustre: Skipped 1 previous similar message [ 4876.278167] Lustre: Skipped 1 previous similar message [ 4880.805253] Lustre: server umount lustre-MDT0000 complete [ 4881.379278] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4881.386841] LustreError: Skipped 3 previous similar messages [ 4882.521384] LustreError: 281842:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760456629 with bad export cookie 953146420286021982 [ 4882.525513] LustreError: 166-1: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4882.526499] LustreError: 281842:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4886.497719] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4886.503869] Lustre: Skipped 1 previous similar message [ 4886.506671] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4886.510307] Lustre: Skipped 2 previous similar messages [ 4888.737108] Lustre: server umount lustre-MDT0001 complete [ 4894.925591] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_hostid [ 4898.980903] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 4903.580832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4906.706947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4909.730280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4912.719461] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4916.007977] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 11:44:21 (1760456661) [ 4919.251811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4919.295818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4919.445438] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4919.466407] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4919.525374] Lustre: lustre-MDT0000: new disk, initializing [ 4919.566517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4919.580406] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4921.616093] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 4926.952248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4926.991147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4927.041562] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4927.072694] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 4927.075943] Lustre: Skipped 1 previous similar message [ 4927.124144] Lustre: lustre-MDT0001: new disk, initializing [ 4927.169285] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4927.184259] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4927.188473] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4929.078365] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 4933.602348] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4935.722373] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4938.260515] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4938.287427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4938.291758] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4938.390969] Lustre: lustre-OST0000: new disk, initializing [ 4938.392578] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 4938.417275] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4940.237420] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all 8 [ 4944.730555] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4946.935322] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4946.938898] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4953.057318] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4953.061949] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4953.068622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4953.071639] Lustre: Skipped 2 previous similar messages [ 4957.256255] Lustre: server umount lustre-MDT0000 complete [ 4957.257750] Lustre: Skipped 1 previous similar message [ 4958.942756] LustreError: 325283:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760456705 with bad export cookie 953146420286023879 [ 4958.946589] LustreError: 326157:0:(osp_object.c:629:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x2:0x0]: rc = -5 [ 4958.948313] LustreError: 166-1: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4958.949751] LustreError: 325283:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4958.959104] LustreError: 326157:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000009fca7bae x1845971528102144/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 264/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' [ 4958.977948] LustreError: 326157:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) lustre-MDT0000-osp-MDT0001: fail to cancel 1 llog-records: rc = -5 [ 4958.982874] LustreError: 326157:0:(llog_cat.c:789:llog_cat_cancel_records()) lustre-MDT0000-osp-MDT0001: fail to cancel 1 of 1 llog-records: rc = -5 [ 4966.880136] Lustre: 328544:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760456707/real 1760456707] req@00000000d4fb3058 x1845971528102336/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1760456713 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4977.862559] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_hostid [ 4980.784263] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 4985.025365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4988.035293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4990.667401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4993.313095] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4994.063927] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 4831 sec ========== 11:45:40 (1760456740) [ 5009.328956] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 5010.263482] Key type lgssc unregistered [ 5010.369301] LNet: 333640:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5010.371675] LNet: Removed LNI 192.168.203.149@tcp [ 5010.662720] Key type .llcrypt unregistered [ 5010.664774] Key type ._llcrypt unregistered