[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 465734355 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003110] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.006008] kvm-guest: setup PV IPIs [ 0.009000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010013] pid_max: default: 32768 minimum: 301 [ 0.011142] LSM: Security Framework initializing [ 0.012061] Yama: becoming mindful. [ 0.013038] SELinux: Initializing. [ 0.014066] *** VALIDATE selinux *** [ 0.023162] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028329] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029149] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030121] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031120] *** VALIDATE tmpfs *** [ 0.033426] *** VALIDATE proc *** [ 0.034222] *** VALIDATE cgroup *** [ 0.035009] *** VALIDATE cgroup2 *** [ 0.036270] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038042] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041026] Spectre V2 : User space: Vulnerable [ 0.042012] Speculative Store Bypass: Vulnerable [ 0.045567] debug: unmapping init [mem 0xffffffffa4259000-0xffffffffa4260fff] [ 0.047254] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048710] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049025] ... version: 2 [ 0.050013] ... bit width: 48 [ 0.051012] ... generic registers: 4 [ 0.052022] ... value mask: 0000ffffffffffff [ 0.053016] ... max period: 00007fffffffffff [ 0.054015] ... fixed-purpose events: 3 [ 0.055012] ... event mask: 000000070000000f [ 0.056301] rcu: Hierarchical SRCU implementation. [ 0.058471] smp: Bringing up secondary CPUs ... [ 0.059585] x86: Booting SMP configuration: [ 0.060028] .... node #0, CPUs: #1 #2 #3 [ 0.064013] smp: Brought up 1 node, 4 CPUs [ 0.066017] smpboot: Max logical packages: 1 [ 0.067012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.143067] node 0 deferred pages initialised in 74ms [ 0.147142] devtmpfs: initialized [ 0.148271] x86/mm: Memory block size: 128MB [ 0.150752] gcov: version magic: 0x41383552 [ 0.152357] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.156159] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.158320] pinctrl core: initialized pinctrl subsystem [ 0.160176] [ 0.160817] ************************************************************* [ 0.163012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.164010] ** ** [ 0.166013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.169013] ** ** [ 0.171010] ** This means that this kernel is built to expose internal ** [ 0.173016] ** IOMMU data structures, which may compromise security on ** [ 0.176014] ** your system. ** [ 0.179014] ** ** [ 0.181011] ** If you see this message and you are not debugging the ** [ 0.183013] ** kernel, report this immediately to your vendor! ** [ 0.185011] ** ** [ 0.187011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.189008] ************************************************************* [ 0.191530] NET: Registered protocol family 16 [ 0.192363] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.194052] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.196041] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.198373] cpuidle: using governor menu [ 0.199624] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.201307] PCI: Using configuration type 1 for base access [ 0.202151] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.209053] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.210267] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.212295] cryptd: max_cpu_qlen set to 1000 [ 0.215146] ACPI: Added _OSI(Module Device) [ 0.216006] ACPI: Added _OSI(Processor Device) [ 0.216898] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.218010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.221024] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.225042] ACPI: Interpreter enabled [ 0.225882] ACPI: PM: (supports S0 S3 S4 S5) [ 0.227008] ACPI: Using IOAPIC for interrupt routing [ 0.228000] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.229231] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.237000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.237000] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.240012] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.242066] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.245353] acpiphp: Slot [2] registered [ 0.246083] acpiphp: Slot [5] registered [ 0.247088] acpiphp: Slot [6] registered [ 0.247999] acpiphp: Slot [7] registered [ 0.250116] acpiphp: Slot [8] registered [ 0.250983] acpiphp: Slot [9] registered [ 0.251056] acpiphp: Slot [10] registered [ 0.252071] acpiphp: Slot [3] registered [ 0.252936] acpiphp: Slot [4] registered [ 0.254076] acpiphp: Slot [11] registered [ 0.254948] acpiphp: Slot [12] registered [ 0.256060] acpiphp: Slot [13] registered [ 0.256971] acpiphp: Slot [14] registered [ 0.258062] acpiphp: Slot [15] registered [ 0.259065] acpiphp: Slot [16] registered [ 0.259918] acpiphp: Slot [17] registered [ 0.261050] acpiphp: Slot [18] registered [ 0.261894] acpiphp: Slot [19] registered [ 0.262080] acpiphp: Slot [20] registered [ 0.262927] acpiphp: Slot [21] registered [ 0.264054] acpiphp: Slot [22] registered [ 0.264894] acpiphp: Slot [23] registered [ 0.266059] acpiphp: Slot [24] registered [ 0.266902] acpiphp: Slot [25] registered [ 0.267049] acpiphp: Slot [26] registered [ 0.267907] acpiphp: Slot [27] registered [ 0.269055] acpiphp: Slot [28] registered [ 0.269988] acpiphp: Slot [29] registered [ 0.271067] acpiphp: Slot [30] registered [ 0.271925] acpiphp: Slot [31] registered [ 0.273057] PCI host bridge to bus 0000:00 [ 0.273836] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.275016] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.276013] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.277014] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.279020] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.281020] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.282132] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.284722] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.285916] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.293013] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.295426] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.297011] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.298008] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.299009] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.301407] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.303593] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.305026] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.306606] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.311014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.320012] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.323013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.328419] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.334015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.341017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.355018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.372161] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.377012] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.384014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.397014] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.404932] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.411018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.417017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.433016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.442404] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.456025] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.461018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.483022] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.493297] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.499014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.504016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.515018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.524473] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.530015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.534025] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.545015] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.556882] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.559273] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.561201] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.562262] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.563118] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.568039] iommu: Default domain type: Passthrough [ 0.569410] SCSI subsystem initialized [ 0.570144] ACPI: bus type USB registered [ 0.572112] usbcore: registered new interface driver usbfs [ 0.574087] usbcore: registered new interface driver hub [ 0.576109] usbcore: registered new device driver usb [ 0.578179] pps_core: LinuxPPS API ver. 1 registered [ 0.580014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.583078] PTP clock support registered [ 0.585115] EDAC MC: Ver: 3.0.0 [ 0.587602] PCI: Using ACPI for IRQ routing [ 0.589315] NetLabel: Initializing [ 0.590000] NetLabel: domain hash size = 128 [ 0.592047] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.594108] NetLabel: unlabeled traffic allowed by default [ 0.597045] vgaarb: loaded [ 0.599291] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.600013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.605162] clocksource: Switched to clocksource kvm-clock [ 0.721588] VFS: Disk quotas dquot_6.6.0 [ 0.723820] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.726196] *** VALIDATE ramfs *** [ 0.727140] *** VALIDATE hugetlbfs *** [ 0.728761] pnp: PnP ACPI init [ 0.731301] pnp: PnP ACPI: found 6 devices [ 0.746812] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.750333] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.752202] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.754106] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.756273] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.758935] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.762194] NET: Registered protocol family 2 [ 0.765034] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.770683] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.774650] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.780721] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.784467] TCP: Hash tables configured (established 65536 bind 65536) [ 0.787885] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.791253] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.794715] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.797808] NET: Registered protocol family 1 [ 0.800518] RPC: Registered named UNIX socket transport module. [ 0.802673] RPC: Registered udp transport module. [ 0.804625] RPC: Registered tcp transport module. [ 0.806414] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.809126] NET: Registered protocol family 44 [ 0.811046] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.813278] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.815701] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.818182] PCI: CLS 0 bytes, default 64 [ 0.820170] Unpacking initramfs... [ 2.225164] debug: unmapping init [mem 0xffff98f4fcc54000-0xffff98f4fffbffff] [ 2.229314] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.231684] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.234139] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.755338] Initialise system trusted keyrings [ 2.756691] Key type blacklist registered [ 2.758300] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.769311] zbud: loaded [ 2.773914] *** VALIDATE nfs *** [ 2.775154] *** VALIDATE nfs4 *** [ 2.776751] pstore: using deflate compression [ 2.780811] Platform Keyring initialized [ 2.895365] NET: Registered protocol family 38 [ 2.897268] Key type asymmetric registered [ 2.898358] Asymmetric key parser 'x509' registered [ 2.899858] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.902807] io scheduler mq-deadline registered [ 2.904226] io scheduler kyber registered [ 2.905517] io scheduler bfq registered [ 2.907190] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.909737] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.912307] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.914808] ACPI: Power Button [PWRF] [ 2.918839] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.924063] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.936401] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.943591] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.959227] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.986485] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.014711] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.019629] Non-volatile memory driver v1.3 [ 3.021271] Linux agpgart interface v0.103 [ 3.053083] virtio_blk virtio1: [vda] 67992 512-byte logical blocks (34.8 MB/33.2 MiB) [ 3.055794] vda: detected capacity change from 0 to 34811904 [ 3.070285] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.073394] vdb: detected capacity change from 0 to 1073741824 [ 3.086637] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.088907] vdc: detected capacity change from 0 to 2621440000 [ 3.104096] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.106788] vdd: detected capacity change from 0 to 2621440000 [ 3.120148] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.123115] vde: detected capacity change from 0 to 4294967296 [ 3.136662] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.140128] vdf: detected capacity change from 0 to 4294967296 [ 3.148563] libphy: Fixed MDIO Bus: probed [ 3.168630] usbcore: registered new interface driver usbserial_generic [ 3.171182] usbserial: USB Serial support registered for generic [ 3.173629] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.178200] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.180184] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.183626] mousedev: PS/2 mouse device common for all mice [ 3.187483] rtc_cmos 00:05: RTC can wake from S4 [ 3.190312] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.195220] rtc_cmos 00:05: registered as rtc0 [ 3.197293] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.200840] intel_pstate: CPU model not supported [ 3.201553] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.207239] hid: raw HID events driver (C) Jiri Kosina [ 3.208539] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.210179] usbcore: registered new interface driver usbhid [ 3.215503] usbhid: USB HID core driver [ 3.217096] drop_monitor: Initializing network drop monitor service [ 3.219182] Initializing XFRM netlink socket [ 3.220755] NET: Registered protocol family 10 [ 3.224042] Segment Routing with IPv6 [ 3.225668] NET: Registered protocol family 17 [ 3.228678] mpls_gso: MPLS GSO support [ 3.234794] RAS: Correctable Errors collector initialized. [ 3.237218] AVX version of gcm_enc/dec engaged. [ 3.238942] AES CTR mode by8 optimization enabled [ 3.318878] sched_clock: Marking stable (3318807582, 0)->(4203004077, -884196495) [ 3.323649] registered taskstats version 1 [ 3.325971] Loading compiled-in X.509 certificates [ 3.328320] zswap: loaded using pool lzo/zbud [ 3.357215] Key type big_key registered [ 3.371761] Key type encrypted registered [ 3.373451] ima: No TPM chip found, activating TPM-bypass! [ 3.375739] ima: Allocated hash algorithm: sha1 [ 3.377432] ima: No architecture policies found [ 3.379173] evm: Initialising EVM extended attributes: [ 3.381122] evm: security.selinux [ 3.382363] evm: security.ima [ 3.383501] evm: security.capability [ 3.384892] evm: HMAC attrs: 0x1 [ 3.387413] rtc_cmos 00:05: setting system clock to 2026-04-14 20:52:21 UTC (1776199941) [ 3.393537] debug: unmapping init [mem 0xffffffffa5203000-0xffffffffa53fffff] [ 3.396196] debug: unmapping init [mem 0xffffffffa3f82000-0xffffffffa4258fff] [ 3.404594] Write protecting the kernel read-only data: 28672k [ 3.407945] debug: unmapping init [mem 0xffffffffa2603000-0xffffffffa27fffff] [ 3.411094] debug: unmapping init [mem 0xffffffffa2f14000-0xffffffffa2ffffff] [ 3.443819] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.452548] systemd[1]: Detected virtualization kvm. [ 3.454203] systemd[1]: Detected architecture x86-64. [ 3.456087] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.482877] systemd[1]: No hostname configured. [ 3.484914] systemd[1]: Set hostname to . [ 3.486718] random: systemd: uninitialized urandom read (16 bytes read) [ 3.489306] systemd[1]: Initializing machine ID from random generator. [ 3.569215] random: ln: uninitialized urandom read (6 bytes read) [ 3.660595] random: systemd: uninitialized urandom read (16 bytes read) [ 3.664087] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.669632] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.675976] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local File Systems. [ OK ] Reached target Slices. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. [ OK ] Reached target Paths. Starting Journal Service... [ OK ] Reached target Initrd Root Device. Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... [ OK ] Reached target Timers. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.349815] device-mapper: uevent: version 1.0.3 [ 4.352129] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.169884] random: fast init done [ 5.203749] virtio_net virtio0 ens2: renamed from eth0 [ 5.275302] scsi host0: ata_piix [ 5.322847] scsi host1: ata_piix [ 5.329349] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.332038] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.011558] dracut-initqueue[581]: RTNETLINK answers: File exists [ 9.890982] random: crng init done [ 9.892297] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.450260] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.564285] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.841988] SELinux: Disabled at runtime. [ 11.906514] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.913311] systemd[1]: Detected virtualization kvm. [ 11.914953] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.453363] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.457105] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.464767] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.468169] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.471945] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.481285] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.485923] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Control Socket. Starting Remount Root and Kernel File Systems... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-getty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-serial\x2dgetty.slice. [ 12.538935] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Mounting Kernel Debug File System... [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting POSIX Message Queue File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on initctl Compatibility Named Pipe. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Apply Kernel Variables... [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Mounting Huge Pages File System... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.949744] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.392200] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.396780] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.550242] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.569605] EDAC sbridge: Ver: 1.1.2 [ 14.848300] Key type dns_resolver registered [ 15.139930] NFS: Registering the id_resolver key type [ 15.141916] Key type id_resolver registered [ 15.143485] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg254-server login: [ 45.396742] libcfs: loading out-of-tree module taints kernel. [ 45.443140] alg: No test for adler32 (adler32-zlib) [ 46.192317] Key type ._llcrypt registered [ 46.194193] Key type .llcrypt registered [ 46.298820] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [ 63.069895] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 64.342464] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 64.894513] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 65.491852] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 65.494650] LNet: Accept secure, port 988 [ 67.169040] Key type lgssc registered [ 68.342350] Lustre: Echo OBD driver; http://www.lustre.org/ [ 83.919686] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 86.524266] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 94.799522] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 100.356164] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 106.500769] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 119.558215] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 131.893251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 131.995206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 133.250206] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 133.285898] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 133.386628] Lustre: lustre-MDT0000: new disk, initializing [ 133.511263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 133.545505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 138.744661] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 153.079556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 153.161593] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 153.220025] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 153.234874] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 153.240039] Lustre: Skipped 1 previous similar message [ 153.340593] Lustre: lustre-MDT0001: new disk, initializing [ 153.421713] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 153.438412] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 153.450455] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 157.877549] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 169.929653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 170.002724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 170.009278] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 170.194456] Lustre: lustre-OST0000: new disk, initializing [ 170.198921] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 170.262519] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 171.083377] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 171.094873] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 174.106933] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 187.074323] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 187.126862] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 187.135577] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 187.222615] Lustre: lustre-OST0001: new disk, initializing [ 187.228267] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 187.304171] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 191.394669] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 193.068804] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 193.080410] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 201.436951] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 207.980222] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 216.580459] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing check_logdir /tmp/testlogs/ [ 221.420652] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing yml_node [ 225.117815] Lustre: DEBUG MARKER: Client: 2.15.8.2 [ 227.701970] Lustre: DEBUG MARKER: MDS: 2.15.8.2 [ 230.342334] Lustre: DEBUG MARKER: OSS: 2.15.8.2 [ 232.006984] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Apr 14 16:56:09 EDT 2026 [ 240.378261] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball 110 [ 241.669153] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 [ 260.069257] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 260.079095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 260.083861] Lustre: Skipped 1 previous similar message [ 262.709739] Lustre: server umount lustre-MDT0000 complete [ 265.188166] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 265.198866] LustreError: Skipped 3 previous similar messages [ 265.881718] LustreError: 6255:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776200203 with bad export cookie 5872558715488644540 [ 265.884739] LustreError: 166-1: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 265.899870] LustreError: 6255:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 270.307232] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 270.309144] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 270.310290] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 270.310297] Lustre: Skipped 3 previous similar messages [ 270.316130] LustreError: Skipped 1 previous similar message [ 270.354541] Lustre: Skipped 3 previous similar messages [ 272.342334] Lustre: server umount lustre-MDT0001 complete [ 276.217867] Lustre: server umount lustre-OST0000 complete [ 279.762944] Lustre: server umount lustre-OST0001 complete [ 285.275277] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [ 292.598463] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 301.558446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 306.919522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 312.340122] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 313.939076] hrtimer: interrupt took 2221439 ns [ 317.815506] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 323.589893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 323.675386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 323.970942] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 324.017938] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 324.114252] Lustre: lustre-MDT0000: new disk, initializing [ 324.263088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 324.289718] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 328.977978] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 339.358340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 339.422942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 339.491291] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 339.530523] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 339.535339] Lustre: Skipped 1 previous similar message [ 339.605230] Lustre: lustre-MDT0001: new disk, initializing [ 339.674622] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 339.706445] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 339.714716] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 343.918149] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 353.038626] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 359.037315] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 365.430482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 365.495666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 365.501722] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 365.767781] Lustre: lustre-OST0000: new disk, initializing [ 365.771281] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 365.848311] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 367.708787] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 367.714962] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 369.936861] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 379.342840] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 383.594732] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 383.829563] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 388.101780] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 388.282540] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 393.708428] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 393.719992] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 393.728224] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 393.730650] Lustre: Skipped 2 previous similar messages [ 395.999586] Lustre: server umount lustre-OST0000 complete [ 401.376280] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 401.384960] LustreError: Skipped 1 previous similar message [ 401.393753] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 401.404486] Lustre: Skipped 1 previous similar message [ 401.414952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 405.817214] Lustre: server umount lustre-MDT0000 complete [ 409.060369] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 409.067417] LustreError: Skipped 1 previous similar message [ 410.020169] LustreError: 14346:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776200348 with bad export cookie 5872558715488645996 [ 410.031159] LustreError: 166-1: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 410.032051] LustreError: 14346:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 413.610579] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 16:59:10 (1776200350) [ 419.324530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 419.690907] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 419.756070] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 423.832554] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 424.931065] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 432.762792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 437.430091] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 445.576797] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 452.374988] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 458.369584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 462.803599] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 470.929966] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 477.155678] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 477.162918] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 477.179284] Lustre: Skipped 1 previous similar message [ 477.184444] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 477.187248] Lustre: Skipped 2 previous similar messages [ 479.207106] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 483.342294] Lustre: server umount lustre-OST0000 complete [ 483.344875] Lustre: Skipped 1 previous similar message [ 489.444127] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 489.449809] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 489.470854] Lustre: Skipped 1 previous similar message [ 489.478771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 489.485354] Lustre: Skipped 3 previous similar messages [ 501.215122] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 501.747097] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 501.770656] LustreError: Skipped 2 previous similar messages [ 506.696604] LustreError: 18741:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776200444 with bad export cookie 5872558715488646850 [ 506.701246] LustreError: 166-1: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 506.704385] LustreError: 18741:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 521.629658] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 525.029098] Key type lgssc unregistered [ 525.299308] LNet: 22427:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 525.310709] LNet: Removed LNI 192.168.202.154@tcp [ 526.293924] Key type .llcrypt unregistered [ 526.299148] Key type ._llcrypt unregistered [ 548.905317] alg: No test for adler32 (adler32-zlib) [ 549.662963] Key type ._llcrypt registered [ 549.666962] Key type .llcrypt registered [ 549.767309] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 550.859785] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 551.145541] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 551.339545] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 551.347148] LNet: Accept secure, port 988 [ 553.023128] Key type lgssc registered [ 553.790271] Lustre: Echo OBD driver; http://www.lustre.org/ [ 561.004064] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 562.488371] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 562.591387] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 566.965761] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 567.782615] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 572.897440] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 574.898598] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 575.328847] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 579.352815] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 586.108287] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 591.259251] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 597.833708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 598.285245] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 602.446787] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 609.008652] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 612.662868] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 621.025190] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 621.033201] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 621.037897] LustreError: Skipped 1 previous similar message [ 621.043600] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 621.053511] Lustre: Skipped 1 previous similar message [ 623.106186] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 623.119708] Lustre: Skipped 1 previous similar message [ 625.520887] Lustre: server umount lustre-OST0000 complete [ 631.776683] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 631.786700] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 631.798466] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 631.805946] Lustre: Skipped 1 previous similar message [ 633.316230] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 634.832525] Lustre: server umount lustre-MDT0000 complete [ 638.304433] LustreError: 23873:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776200576 with bad export cookie 1081646981538146621 [ 638.306614] LustreError: 166-1: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 638.314872] LustreError: 23873:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 638.578902] Lustre: server umount lustre-MDT0001 complete [ 652.070086] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 655.023562] Key type lgssc unregistered [ 655.403334] LNet: 27890:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 655.426886] LNet: Removed LNI 192.168.202.154@tcp [ 656.584164] Key type .llcrypt unregistered [ 656.588815] Key type ._llcrypt unregistered [ 685.658980] alg: No test for adler32 (adler32-zlib) [ 686.430412] Key type ._llcrypt registered [ 686.436086] Key type .llcrypt registered [ 686.616650] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 687.997980] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 689.132842] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 689.412354] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 689.420414] LNet: Accept secure, port 988 [ 691.144619] Key type lgssc registered [ 692.348807] Lustre: Echo OBD driver; http://www.lustre.org/ [ 700.005805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 701.496317] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 701.537487] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 705.457432] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 707.042455] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 712.167425] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 713.620682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 713.929514] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 717.969129] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 725.050196] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 730.412303] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 743.698688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 744.111031] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 747.848681] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 752.300784] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 754.264527] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 772.075491] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 772.086162] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 772.103954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 774.117777] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 774.119488] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 774.129316] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 774.132577] Lustre: Skipped 1 previous similar message [ 776.819706] Lustre: server umount lustre-OST0000 complete [ 780.768954] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 780.789304] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 780.807983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 784.355520] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 784.356728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 784.400056] Lustre: Skipped 1 previous similar message [ 789.473766] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 789.483956] Lustre: Skipped 1 previous similar message [ 795.103159] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 795.274482] Lustre: server umount lustre-MDT0000 complete [ 795.619379] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 795.642807] LustreError: Skipped 1 previous similar message [ 799.013569] LustreError: 29355:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776200737 with bad export cookie 2038942675773625500 [ 799.022238] LustreError: 166-1: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 799.030990] LustreError: 29355:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 799.357398] Lustre: server umount lustre-MDT0001 complete [ 812.218269] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 815.079625] Key type lgssc unregistered [ 815.374875] LNet: 33398:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 815.394258] LNet: Removed LNI 192.168.202.154@tcp [ 816.119419] Key type .llcrypt unregistered [ 816.121903] Key type ._llcrypt unregistered [ 837.141909] alg: No test for adler32 (adler32-zlib) [ 837.899382] Key type ._llcrypt registered [ 837.901658] Key type .llcrypt registered [ 837.987076] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 839.455856] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 839.827498] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 840.072530] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 840.075595] LNet: Accept secure, port 988 [ 841.695157] Key type lgssc registered [ 842.638752] Lustre: Echo OBD driver; http://www.lustre.org/ [ 850.362928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 851.845147] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 851.913334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 855.909665] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 856.344245] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 861.156066] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 865.321482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 865.701549] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 869.822406] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 878.308984] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 884.476786] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 890.745774] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 891.062051] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 895.116769] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 901.304538] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 902.122287] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 911.843561] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 911.851656] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 911.869343] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 911.884976] Lustre: Skipped 1 previous similar message [ 916.973188] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 916.983903] Lustre: Skipped 1 previous similar message [ 917.242193] Lustre: server umount lustre-OST0000 complete [ 922.080368] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 922.083779] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 922.106200] Lustre: Skipped 1 previous similar message [ 922.109406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 927.201502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 927.204319] Lustre: Skipped 1 previous similar message [ 928.078450] Lustre: server umount lustre-MDT0000 complete [ 932.320417] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 932.540155] LustreError: 34844:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776200870 with bad export cookie 13807820791699042378 [ 932.544371] LustreError: 166-1: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 932.556619] LustreError: 34844:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 933.075874] Lustre: server umount lustre-MDT0001 complete [ 944.851915] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 947.239632] Key type lgssc unregistered [ 947.587193] LNet: 38522:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 947.598573] LNet: Removed LNI 192.168.202.154@tcp [ 948.403638] Key type .llcrypt unregistered [ 948.405613] Key type ._llcrypt unregistered [ 975.079626] alg: No test for adler32 (adler32-zlib) [ 975.852437] Key type ._llcrypt registered [ 975.854440] Key type .llcrypt registered [ 976.063121] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 977.852512] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 978.535228] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 978.761401] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 978.772237] LNet: Accept secure, port 988 [ 980.679812] Key type lgssc registered [ 982.187570] Lustre: Echo OBD driver; http://www.lustre.org/ [ 992.074813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 993.522789] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 993.564323] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 997.185786] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 998.882877] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1004.000299] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1004.050272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1004.262445] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1007.850758] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 1014.592896] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1020.462713] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1026.532219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1026.876305] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1030.773746] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 1038.327105] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1043.244225] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 1048.552396] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1048.559908] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1048.573106] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1048.588502] Lustre: Skipped 1 previous similar message [ 1048.602153] Lustre: Skipped 1 previous similar message [ 1049.522980] Lustre: lustre-OST0000: Not available for connect from 192.168.202.54@tcp (stopping) [ 1053.667124] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1053.673492] Lustre: Skipped 1 previous similar message [ 1054.543835] Lustre: server umount lustre-OST0000 complete [ 1076.192247] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1076.199246] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1076.214211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1079.267107] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1080.921984] Lustre: server umount lustre-MDT0000 complete [ 1084.370600] LustreError: 39988:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776201022 with bad export cookie 12292850295194073749 [ 1084.371763] LustreError: 166-1: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1084.379643] LustreError: 39988:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1090.821428] Lustre: server umount lustre-MDT0001 complete [ 1105.131590] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 1107.702460] Key type lgssc unregistered [ 1107.963811] LNet: 43837:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1107.970053] LNet: Removed LNI 192.168.202.154@tcp [ 1108.530585] Key type .llcrypt unregistered [ 1108.533577] Key type ._llcrypt unregistered [ 1134.671675] alg: No test for adler32 (adler32-zlib) [ 1135.428745] Key type ._llcrypt registered [ 1135.430802] Key type .llcrypt registered [ 1135.551673] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 1136.847445] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1137.512164] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 1137.861986] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 1137.865811] LNet: Accept secure, port 988 [ 1139.559602] Key type lgssc registered [ 1140.870406] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1149.410469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1150.949486] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1151.005575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1155.641936] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 1156.065425] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1161.185573] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1163.599772] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1163.976793] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1168.106671] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 1175.876117] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1181.397046] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1187.938170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1188.348850] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1193.060721] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 1200.607571] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1201.659840] Lustre: lustre-OST0000: deleting orphan objects from 0x0:163 to 0x0:193 [ 1215.458501] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1215.468964] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1215.483688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1216.937538] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.54@tcp (stopping) [ 1216.998627] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1217.026585] Lustre: Skipped 1 previous similar message [ 1218.459646] Lustre: server umount lustre-MDT0000 complete [ 1221.891823] LustreError: 45312:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776201160 with bad export cookie 1514254820572107286 [ 1221.893978] LustreError: 166-1: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1221.898774] LustreError: 45312:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1222.040912] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1222.082504] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.54@tcp (stopping) [ 1222.086152] Lustre: Skipped 3 previous similar messages [ 1222.112643] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1226.147692] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1226.147757] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1226.155611] Lustre: Skipped 1 previous similar message [ 1226.181630] LustreError: Skipped 1 previous similar message [ 1228.243509] Lustre: server umount lustre-MDT0001 complete [ 1249.236782] Lustre: server umount lustre-OST0000 complete [ 1265.637417] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 1267.772319] Key type lgssc unregistered [ 1267.987803] LNet: 49317:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1267.994408] LNet: Removed LNI 192.168.202.154@tcp [ 1268.597694] Key type .llcrypt unregistered [ 1268.600560] Key type ._llcrypt unregistered [ 1287.795596] alg: No test for adler32 (adler32-zlib) [ 1288.550410] Key type ._llcrypt registered [ 1288.553469] Key type .llcrypt registered [ 1288.638948] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 1289.639902] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1289.964662] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 1290.135883] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 1290.138617] LNet: Accept secure, port 988 [ 1291.735183] Key type lgssc registered [ 1292.792351] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1299.006780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1316.575537] LustreError: 50752:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.154@tcp: failed processing log, type 1: rc = -5 [ 1347.296379] LustreError: 50752:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.154@tcp: failed processing log, type 4: rc = -110 [ 1378.016134] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1382.744968] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 1391.219756] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1430.775806] Lustre: server umount lustre-OST0000 complete [ 1445.764501] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 1448.257366] Key type lgssc unregistered [ 1448.611509] LNet: 52432:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1448.620306] LNet: Removed LNI 192.168.202.154@tcp [ 1449.304842] Key type .llcrypt unregistered [ 1449.307376] Key type ._llcrypt unregistered [ 1472.608917] alg: No test for adler32 (adler32-zlib) [ 1473.371544] Key type ._llcrypt registered [ 1473.381039] Key type .llcrypt registered [ 1473.528414] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 1474.798158] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1475.200239] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 1475.440569] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 1475.444507] LNet: Accept secure, port 988 [ 1477.103974] Key type lgssc registered [ 1478.255892] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1485.828515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1487.221680] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1487.273430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1491.082443] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 1492.449045] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1497.576050] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1499.180279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1499.521048] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1504.655206] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 1513.132109] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1519.640819] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1526.250881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1526.600715] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1531.642673] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 1537.923875] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:225 [ 1540.245377] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1543.141260] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1543.146929] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1543.166209] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1543.188225] Lustre: Skipped 1 previous similar message [ 1543.201807] Lustre: Skipped 1 previous similar message [ 1544.831040] Lustre: server umount lustre-OST0000 complete [ 1550.815989] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1550.826466] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1550.837626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1553.378778] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1553.384029] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1553.408640] Lustre: Skipped 1 previous similar message [ 1554.803601] Lustre: server umount lustre-MDT0000 complete [ 1558.185384] LustreError: 53875:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776201496 with bad export cookie 4241495431037716547 [ 1558.191834] LustreError: 166-1: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1558.448786] Lustre: server umount lustre-MDT0001 complete [ 1569.992485] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 1572.487421] Key type lgssc unregistered [ 1572.739747] LNet: 57548:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1572.748671] LNet: Removed LNI 192.168.202.154@tcp [ 1573.426666] Key type .llcrypt unregistered [ 1573.428931] Key type ._llcrypt unregistered [ 1602.167666] alg: No test for adler32 (adler32-zlib) [ 1602.927691] Key type ._llcrypt registered [ 1602.933619] Key type .llcrypt registered [ 1603.172833] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 1604.619897] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1604.967177] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 1605.196367] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 1605.200483] LNet: Accept secure, port 988 [ 1606.951123] Key type lgssc registered [ 1608.022666] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1615.531964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1632.223487] LustreError: 59004:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.154@tcp: failed processing log, type 1: rc = -5 [ 1663.967358] LustreError: 59004:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.154@tcp: failed processing log, type 4: rc = -110 [ 1693.727309] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1697.636880] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 1704.680714] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1711.652569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1716.703777] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1716.811493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1716.873848] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:257 [ 1721.844657] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1722.201118] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 1726.947022] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1730.964599] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1731.276034] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1736.092786] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 1744.756078] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1751.384593] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1757.178046] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1757.202281] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1757.225141] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1758.185623] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1758.199049] Lustre: Skipped 1 previous similar message [ 1759.897465] Lustre: server umount lustre-OST0000 complete [ 1782.753350] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1782.761133] LustreError: Skipped 1 previous similar message [ 1782.767173] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1782.795961] Lustre: Skipped 1 previous similar message [ 1782.805550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1782.811452] Lustre: Skipped 1 previous similar message [ 1784.803079] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1784.821427] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1784.840536] Lustre: Skipped 1 previous similar message [ 1788.442382] Lustre: server umount lustre-MDT0000 complete [ 1789.920223] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1789.939882] LustreError: Skipped 1 previous similar message [ 1792.450775] LustreError: 59012:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776201730 with bad export cookie 8214844278878829508 [ 1792.451288] LustreError: 166-1: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1792.458613] LustreError: 59012:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1798.918667] Lustre: server umount lustre-MDT0001 complete [ 1810.824918] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 1813.584384] Key type lgssc unregistered [ 1813.888064] LNet: 62872:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1813.895333] LNet: Removed LNI 192.168.202.154@tcp [ 1814.580391] Key type .llcrypt unregistered [ 1814.583483] Key type ._llcrypt unregistered [ 1837.383344] alg: No test for adler32 (adler32-zlib) [ 1838.138746] Key type ._llcrypt registered [ 1838.147220] Key type .llcrypt registered [ 1838.245893] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 1839.171220] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1839.834030] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 1840.027632] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 1840.032909] LNet: Accept secure, port 988 [ 1841.735128] Key type lgssc registered [ 1842.725229] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1849.810274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1851.290776] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1851.340883] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1855.730930] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 1856.481206] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1861.607411] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1863.229591] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1863.773839] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1868.255971] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 1877.337458] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1884.517520] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1895.167446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1895.859901] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1901.003302] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:289 [ 1902.208108] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 1911.978434] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1929.703188] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1929.706668] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1929.718304] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1934.822851] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1934.830804] Lustre: Skipped 2 previous similar messages [ 1935.002862] Lustre: server umount lustre-OST0000 complete [ 1939.938772] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1939.942805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1939.958854] Lustre: Skipped 2 previous similar messages [ 1939.969063] Lustre: Skipped 1 previous similar message [ 1944.608941] Lustre: server umount lustre-MDT0000 complete [ 1945.057612] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1945.078711] LustreError: Skipped 1 previous similar message [ 1948.312776] LustreError: 67209:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776201886 with bad export cookie 12681506763385669060 [ 1948.318639] LustreError: 166-1: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1948.322120] LustreError: 67209:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1948.622214] Lustre: server umount lustre-MDT0001 complete [ 1964.098545] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 1967.049650] Key type lgssc unregistered [ 1967.295932] LNet: 68014:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1967.302903] LNet: Removed LNI 192.168.202.154@tcp [ 1967.927723] Key type .llcrypt unregistered [ 1967.929596] Key type ._llcrypt unregistered [ 2000.849908] alg: No test for adler32 (adler32-zlib) [ 2001.624321] Key type ._llcrypt registered [ 2001.630834] Key type .llcrypt registered [ 2001.820640] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 2003.418061] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2003.785849] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 2004.040704] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 2004.052682] LNet: Accept secure, port 988 [ 2005.751134] Key type lgssc registered [ 2006.964673] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2016.423461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2018.029936] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2018.082320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2023.396708] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2024.308923] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 2028.528550] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2033.636585] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2034.409515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2034.851438] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2039.611672] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 2048.444283] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2054.786518] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2062.246334] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2062.735364] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2067.906274] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 2068.914802] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:321 [ 2075.574152] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2081.152554] Lustre: Failing over lustre-MDT0000 [ 2081.349190] Lustre: server umount lustre-MDT0000 complete [ 2082.213144] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2084.323299] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2084.334285] Lustre: Skipped 2 previous similar messages [ 2090.349640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2090.532608] LustreError: 11-0: MGC192.168.202.154@tcp: operation mgs_target_reg to node 0@lo failed: rc = -107 [ 2090.536407] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2090.536935] LustreError: 166-1: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2090.552605] Lustre: Evicted from MGS (at 192.168.202.154@tcp) after server handle changed from 0x27d69534b9b78203 to 0x27d69534b9b785ec [ 2090.580267] LustreError: Skipped 10 previous similar messages [ 2090.608266] Lustre: MGC192.168.202.154@tcp: Connection restored to (at 0@lo) [ 2090.749456] Lustre: *** cfs_fail_loc=135, val=0*** [ 2090.902520] BUG: spinlock bad magic on CPU#0, llog_process_th/72931 [ 2090.909915] lock: 0xffff98f57da044c0, .magic: 00000000, .owner: /-1, .owner_cpu: 0 [ 2090.916887] CPU: 0 PID: 72931 Comm: llog_process_th Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 [ 2090.929384] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 2090.936936] Call Trace: [ 2090.939032] ? dump_stack+0xbb/0x10e [ 2090.943442] ? spin_dump.cold.3+0x32/0x65 [ 2090.945338] ? do_raw_spin_lock+0xce/0x190 [ 2090.947168] ? _raw_spin_lock+0x21/0x30 [ 2090.949824] ? mdd_changelog_fini+0x45/0x3a0 [mdd] [ 2090.952366] ? mdd_process_config+0x368/0x9c0 [mdd] [ 2090.954162] ? mdt_stack_fini+0x5de/0x1200 [mdt] [ 2090.956249] ? mdt_init0+0x650/0x1750 [mdt] [ 2090.959204] ? mdt_device_alloc+0x118/0x170 [mdt] [ 2090.960890] ? obd_setup+0x196/0x3e0 [obdclass] [ 2090.963563] ? class_setup+0x6ee/0x9e0 [obdclass] [ 2090.965239] ? class_process_config+0x17c7/0x2b90 [obdclass] [ 2090.974770] ? class_config_llog_handler+0xb73/0x1ab0 [obdclass] [ 2090.980581] ? llog_process_thread+0x138f/0x2270 [obdclass] [ 2090.984817] ? mgs_key_init+0xdd/0x110 [mgs] [ 2090.986270] ? keys_fill+0x10d/0x1b0 [obdclass] [ 2090.997649] ? lu_context_init+0xc8/0x240 [obdclass] [ 2091.000621] ? llog_process+0x40/0x40 [obdclass] [ 2091.005529] ? llog_process_thread_daemonize+0xe1/0x150 [obdclass] [ 2091.008855] ? kthread+0x1d1/0x200 [ 2091.015245] ? set_kthread_struct+0x70/0x70 [ 2091.017651] ? ret_from_fork+0x1f/0x30 [ 2091.024808] LustreError: 72931:0:(obd_config.c:774:class_setup()) setup lustre-MDT0000 failed (-2) [ 2091.034745] LustreError: 72931:0:(obd_config.c:1999:class_config_llog_handler()) MGC192.168.202.154@tcp: cfg command failed: rc = -2 [ 2091.040590] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2091.040590] [ 2091.047538] LustreError: 15c-8: MGC192.168.202.154@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2091.059421] LustreError: 72918:0:(obd_mount_server.c:1425:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2091.066090] LustreError: 72918:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -2 [ 2091.072621] LustreError: 72918:0:(obd_config.c:827:class_cleanup()) Device 5 not setup [ 2091.110045] Lustre: server umount lustre-MDT0000 complete [ 2091.112172] LustreError: 72918:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2098.488470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2098.745140] LustreError: 166-1: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2098.746110] LustreError: 11-0: MGC192.168.202.154@tcp: operation mgs_target_reg to node 0@lo failed: rc = -107 [ 2098.776928] LustreError: 69763:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2098.779050] Lustre: Evicted from MGS (at 192.168.202.154@tcp) after server handle changed from 0x27d69534b9b785ec to 0x27d69534b9b78671 [ 2098.782506] Lustre: MGC192.168.202.154@tcp: Connection restored to (at 0@lo) [ 2099.101501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2099.158643] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2100.620761] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2104.311630] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 2104.345311] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2104.419442] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:353 [ 2104.749398] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 2115.041846] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2115.051288] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2115.075216] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2117.089599] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2117.105246] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2117.137608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2119.655555] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2119.673348] Lustre: Skipped 1 previous similar message [ 2121.191801] Lustre: server umount lustre-OST0000 complete [ 2129.888142] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2129.890797] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2129.906491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2129.923378] Lustre: Skipped 1 previous similar message [ 2129.933857] Lustre: Skipped 1 previous similar message [ 2131.977754] Lustre: server umount lustre-MDT0000 complete [ 2135.010410] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2135.027981] LustreError: Skipped 9 previous similar messages [ 2136.044484] LustreError: 69751:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776202074 with bad export cookie 2870645866181920369 [ 2136.054770] LustreError: 166-1: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2136.059404] LustreError: 69751:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2136.343206] Lustre: server umount lustre-MDT0001 complete [ 2151.292826] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 2153.961428] Key type lgssc unregistered [ 2154.180996] LNet: 74817:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2154.190773] LNet: Removed LNI 192.168.202.154@tcp [ 2154.910948] Key type .llcrypt unregistered [ 2154.912756] Key type ._llcrypt unregistered [ 2175.867245] alg: No test for adler32 (adler32-zlib) [ 2176.623778] Key type ._llcrypt registered [ 2176.626495] Key type .llcrypt registered [ 2176.713454] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 2177.618768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2177.995663] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 2178.172721] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 2178.175984] LNet: Accept secure, port 988 [ 2179.820142] Key type lgssc registered [ 2180.707606] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2188.132211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2189.424811] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2189.477746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2194.346309] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 2194.915280] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2200.046398] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2204.424943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2205.159909] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2205.613081] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2211.325824] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 2219.228953] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2225.427468] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2231.573526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2231.884050] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2235.738140] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 2240.123527] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:385 [ 2243.569776] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2249.666327] Lustre: Failing over lustre-MDT0000 [ 2249.803851] Lustre: server umount lustre-MDT0000 complete [ 2250.644663] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2250.723751] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2250.733986] Lustre: Skipped 1 previous similar message [ 2255.753571] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2255.764949] LustreError: Skipped 3 previous similar messages [ 2257.887993] Lustre: 75593:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776202188/real 1776202188] req@000000003e810df8 x1862482909998848/t0(0) o400->MGC192.168.202.154@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776202195 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2257.920864] LustreError: 166-1: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2258.189945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2263.947845] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2263.958657] LustreError: Skipped 11 previous similar messages [ 2264.112854] Lustre: Evicted from MGS (at 192.168.202.154@tcp) after server handle changed from 0x7e9040e000666d0c to 0x7e9040e0006670f5 [ 2264.141721] Lustre: MGC192.168.202.154@tcp: Connection restored to 192.168.202.154@tcp (at 0@lo) [ 2264.328469] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2264.442062] LustreError: 79439:0:(obd_config.c:774:class_setup()) setup lustre-MDT0000 failed (-9) [ 2264.448794] LustreError: 79439:0:(obd_config.c:1999:class_config_llog_handler()) MGC192.168.202.154@tcp: cfg command failed: rc = -9 [ 2264.455542] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2264.455542] [ 2264.465564] LustreError: 15c-8: MGC192.168.202.154@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2264.477630] LustreError: 79426:0:(obd_mount_server.c:1425:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2264.486512] LustreError: 79426:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -9 [ 2264.492921] LustreError: 79426:0:(obd_config.c:827:class_cleanup()) Device 5 not setup [ 2264.500227] LustreError: 76263:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776202202 with bad export cookie 9119860576248819957 [ 2264.508262] LustreError: 166-1: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2264.513898] LustreError: 76263:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2264.548487] Lustre: server umount lustre-MDT0000 complete [ 2264.552776] LustreError: 79426:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2272.958912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2276.838587] Lustre: Evicted from MGS (at 192.168.202.154@tcp) after server handle changed from 0x7e9040e0006670f5 to 0x7e9040e0006671ab [ 2276.863203] Lustre: MGC192.168.202.154@tcp: Connection restored to 192.168.202.154@tcp (at 0@lo) [ 2277.231823] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2277.289192] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2279.316574] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2282.485020] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 2282.512910] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2282.571596] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 2282.577716] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:417 [ 2291.688487] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2291.715743] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2291.738895] Lustre: Skipped 1 previous similar message [ 2291.743622] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2292.719428] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2297.410540] Lustre: server umount lustre-OST0000 complete [ 2302.944690] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2302.952638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2302.955529] Lustre: Skipped 1 previous similar message [ 2302.967988] Lustre: Skipped 3 previous similar messages [ 2308.066843] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2308.073943] Lustre: Skipped 1 previous similar message [ 2313.188454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2313.191412] Lustre: Skipped 1 previous similar message [ 2315.231125] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2315.410537] Lustre: server umount lustre-MDT0000 complete [ 2318.309661] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2318.340213] LustreError: Skipped 18 previous similar messages [ 2319.273771] LustreError: 76262:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776202257 with bad export cookie 9119860576248820139 [ 2319.274440] LustreError: 166-1: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2319.286382] LustreError: 76262:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2319.728893] Lustre: server umount lustre-MDT0001 complete [ 2332.758388] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 2335.303202] Key type lgssc unregistered [ 2335.638651] LNet: 81325:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2335.663539] LNet: Removed LNI 192.168.202.154@tcp [ 2336.540371] Key type .llcrypt unregistered [ 2336.542771] Key type ._llcrypt unregistered [ 2358.007687] alg: No test for adler32 (adler32-zlib) [ 2358.760919] Key type ._llcrypt registered [ 2358.765881] Key type .llcrypt registered [ 2358.868817] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 2359.717623] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2360.394518] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 2360.662310] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 2360.671326] LNet: Accept secure, port 988 [ 2362.415192] Key type lgssc registered [ 2363.514875] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2371.862746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2373.362273] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2373.405674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2377.859736] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 2378.726630] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2383.846187] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2386.555623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2386.916555] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2391.070982] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 2399.982461] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2405.501658] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2412.299980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2412.652657] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2416.602557] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 2423.538955] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2423.923896] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:449 [ 2432.482139] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2432.492280] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2432.510151] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2434.530592] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2434.534960] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2434.564582] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2437.881917] Lustre: server umount lustre-OST0000 complete [ 2443.235921] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2443.249782] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2443.269462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2443.272276] Lustre: Skipped 1 previous similar message [ 2448.817840] Lustre: server umount lustre-MDT0000 complete [ 2449.911489] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2449.936141] LustreError: Skipped 1 previous similar message [ 2453.752222] LustreError: 83575:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776202391 with bad export cookie 11970696498071481462 [ 2453.770569] LustreError: 166-1: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2454.192254] Lustre: server umount lustre-MDT0001 complete [ 2467.945105] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 2471.037737] Key type lgssc unregistered [ 2471.321958] LNet: 86445:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2471.348346] LNet: Removed LNI 192.168.202.154@tcp [ 2472.236876] Key type .llcrypt unregistered [ 2472.239266] Key type ._llcrypt unregistered [ 2497.233241] alg: No test for adler32 (adler32-zlib) [ 2497.987803] Key type ._llcrypt registered [ 2497.990364] Key type .llcrypt registered [ 2498.173247] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 2499.674678] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2500.702807] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 2500.958405] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 2500.968834] LNet: Accept secure, port 988 [ 2502.683529] Key type lgssc registered [ 2503.771165] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2513.895188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2515.577815] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2515.642038] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2520.978748] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 2521.057705] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2526.181400] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2530.936848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2531.298332] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2531.550466] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2536.446343] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 2546.056273] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2553.119661] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2559.663636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2560.026808] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2564.915907] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 2566.189120] Lustre: lustre-OST0000: deleting orphan objects from 0x0:451 to 0x0:481 [ 2572.697822] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2580.684909] Lustre: server umount lustre-OST0000 complete [ 2586.594432] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2586.605791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2586.611436] Lustre: Skipped 1 previous similar message [ 2588.639216] Lustre: 87273:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776202519/real 1776202519] req@00000000dc86e4ae x1862483248690368/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1776202526 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2588.663927] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2588.678385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2588.688057] Lustre: Skipped 2 previous similar messages [ 2589.727536] Lustre: 87272:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776202520/real 1776202520] req@0000000036c3332c x1862483248690496/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1776202527 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 2590.900349] Lustre: server umount lustre-MDT0000 complete [ 2593.761312] Lustre: 87273:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776202524/real 1776202524] req@00000000cf3e8131 x1862483248691328/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1776202531 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2593.771126] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2593.792183] Lustre: 87273:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2593.814694] LustreError: Skipped 1 previous similar message [ 2594.476806] LustreError: 87941:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776202532 with bad export cookie 14132944294350181231 [ 2594.482581] LustreError: 166-1: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2594.494505] LustreError: 87941:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2594.851978] Lustre: server umount lustre-MDT0001 complete [ 2606.196696] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 2608.513744] Key type lgssc unregistered [ 2608.766593] LNet: 91616:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2608.774817] LNet: Removed LNI 192.168.202.154@tcp [ 2609.378346] Key type .llcrypt unregistered [ 2609.379590] Key type ._llcrypt unregistered [ 2631.009853] alg: No test for adler32 (adler32-zlib) [ 2631.765732] Key type ._llcrypt registered [ 2631.772595] Key type .llcrypt registered [ 2631.915815] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 2633.131275] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2633.422767] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 2633.601744] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 2633.606547] LNet: Accept secure, port 988 [ 2635.255128] Key type lgssc registered [ 2636.419951] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2644.373217] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2645.964589] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2646.067309] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2651.109388] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2651.674564] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 2656.225467] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2660.686091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2661.174751] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2666.922647] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 2677.721411] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2685.511632] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2693.630658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2694.103237] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2697.216798] Lustre: lustre-OST0000: deleting orphan objects from 0x0:451 to 0x0:513 [ 2699.733842] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 2708.187657] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2725.487635] Lustre: server umount lustre-OST0000 complete [ 2732.002085] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2732.005494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2732.012586] Lustre: Skipped 1 previous similar message [ 2733.023559] Lustre: 92391:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776202664/real 1776202664] req@00000000ddbd9d09 x1862483387105024/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1776202671 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 2733.062509] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2733.091257] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2733.099488] Lustre: Skipped 2 previous similar messages [ 2735.073335] Lustre: 95951:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776202667/real 1776202667] req@000000007fe1983f x1862483387105728/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1776202673 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2735.091827] Lustre: 95951:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2737.324751] Lustre: server umount lustre-MDT0000 complete [ 2738.152417] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2738.173365] LustreError: Skipped 1 previous similar message [ 2739.168328] Lustre: 92388:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776202670/real 1776202670] req@0000000023678e10 x1862483387105920/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1776202677 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2740.820506] LustreError: 93058:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776202678 with bad export cookie 13800242247750670427 [ 2740.821534] LustreError: 166-1: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2740.838239] LustreError: 93058:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2741.081772] Lustre: server umount lustre-MDT0001 complete [ 2754.241476] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 2756.980835] Key type lgssc unregistered [ 2757.254471] LNet: 96756:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2757.268063] LNet: Removed LNI 192.168.202.154@tcp [ 2758.009126] Key type .llcrypt unregistered [ 2758.011782] Key type ._llcrypt unregistered [ 2780.606842] alg: No test for adler32 (adler32-zlib) [ 2781.368481] Key type ._llcrypt registered [ 2781.370159] Key type .llcrypt registered [ 2781.572188] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 2782.868522] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2783.336267] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 2783.545554] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 2783.558229] LNet: Accept secure, port 988 [ 2785.240205] Key type lgssc registered [ 2786.514246] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2794.557862] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2811.167469] LustreError: 98192:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.154@tcp: failed processing log, type 1: rc = -5 [ 2842.911411] LustreError: 98192:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.154@tcp: failed processing log, type 4: rc = -110 [ 2872.671246] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2878.066664] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 2887.168853] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2893.671920] Lustre: server umount lustre-OST0000 complete [ 2902.722825] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 17:40:39 (1776202839) [ 2909.362749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2909.775447] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2909.817284] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2914.403927] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 2915.299605] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2920.417712] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2923.500728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2923.968943] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2928.900605] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 2938.493314] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2944.212588] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2950.902646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2951.205719] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2956.390186] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 2960.454939] Lustre: lustre-OST0000: deleting orphan objects from 0x0:516 to 0x0:545 [ 2965.167605] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2980.846373] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2980.857185] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2980.870912] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2980.882145] Lustre: Skipped 1 previous similar message [ 2982.159571] Lustre: server umount lustre-OST0000 complete [ 2990.561446] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2990.571802] LustreError: Skipped 1 previous similar message [ 2990.580690] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2990.610851] Lustre: Skipped 1 previous similar message [ 2990.632859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2993.062750] Lustre: server umount lustre-MDT0000 complete [ 2996.195768] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2996.229045] LustreError: Skipped 1 previous similar message [ 2998.581916] LustreError: 99776:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776202936 with bad export cookie 12012259073900167584 [ 2998.586833] LustreError: 166-1: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2998.590330] LustreError: 99776:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2998.962937] Lustre: server umount lustre-MDT0001 complete [ 3010.403413] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 3013.513977] Key type lgssc unregistered [ 3013.747563] LNet: 103581:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3013.751185] LNet: Removed LNI 192.168.202.154@tcp [ 3014.464481] Key type .llcrypt unregistered [ 3014.467122] Key type ._llcrypt unregistered [ 3036.697836] alg: No test for adler32 (adler32-zlib) [ 3037.449262] Key type ._llcrypt registered [ 3037.451449] Key type .llcrypt registered [ 3037.559708] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 3038.580470] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3039.259878] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 3039.453414] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 3039.457584] LNet: Accept secure, port 988 [ 3041.144596] Key type lgssc registered [ 3042.271357] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3050.150851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3051.662028] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3051.719281] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3056.107044] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 3057.121943] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3062.242602] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3064.285979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3064.791172] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3069.201895] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 3076.314235] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3081.952363] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3088.254122] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3088.561107] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3093.807564] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 3102.916410] Lustre: lustre-OST0000: deleting orphan objects from 0x0:516 to 0x0:577 [ 3104.082220] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3118.562694] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3118.565630] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3118.591356] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3118.596344] Lustre: Skipped 1 previous similar message [ 3123.687465] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3123.704306] Lustre: Skipped 1 previous similar message [ 3124.609522] Lustre: server umount lustre-OST0000 complete [ 3131.360521] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3131.374507] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3131.388284] Lustre: Skipped 1 previous similar message [ 3131.392936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3133.922469] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3133.927262] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3139.042391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3139.047961] Lustre: Skipped 2 previous similar messages [ 3143.135375] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3143.359543] Lustre: server umount lustre-MDT0000 complete [ 3144.173074] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3144.195483] LustreError: Skipped 1 previous similar message [ 3147.558931] LustreError: 105025:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776203085 with bad export cookie 9374178739669561964 [ 3147.564281] LustreError: 166-1: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3147.570861] LustreError: 105025:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3147.902710] Lustre: server umount lustre-MDT0001 complete [ 3160.703945] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 3163.817246] Key type lgssc unregistered [ 3164.100858] LNet: 108724:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3164.123772] LNet: Removed LNI 192.168.202.154@tcp [ 3165.169880] Key type .llcrypt unregistered [ 3165.174362] Key type ._llcrypt unregistered [ 3182.925794] alg: No test for adler32 (adler32-zlib) [ 3183.677616] Key type ._llcrypt registered [ 3183.679874] Key type .llcrypt registered [ 3183.786930] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 3185.009414] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3185.576201] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 3185.772045] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 3185.779430] LNet: Accept secure, port 988 [ 3187.426171] Key type lgssc registered [ 3188.418251] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3196.037922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3213.729679] LustreError: 110038:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.154@tcp: failed processing log, type 1: rc = -5 [ 3244.447758] LustreError: 110038:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.154@tcp: failed processing log, type 4: rc = -110 [ 3275.295393] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3280.075173] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 3287.813491] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3293.157506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3296.936113] LustreError: 15c-8: MGC192.168.202.154@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3296.947135] LustreError: 111023:0:(obd_mount_server.c:1425:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3296.952356] LustreError: 111023:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -2 [ 3296.961407] LustreError: 111023:0:(obd_mount_server.c:1644:server_put_super()) no obd lustre-MDT0000 [ 3296.967529] LustreError: 110045:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776203235 with bad export cookie 10384626307227990386 [ 3296.975694] LustreError: 166-1: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3297.122755] Lustre: server umount lustre-MDT0000 complete [ 3297.124794] LustreError: 111023:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3304.840907] Lustre: server umount lustre-OST0000 complete [ 3314.236951] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [ 3323.276863] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 3332.514105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3339.446604] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3346.982277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3354.691358] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3364.083786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3364.155817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3364.370227] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3364.409651] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3364.507733] Lustre: lustre-MDT0000: new disk, initializing [ 3364.606635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3364.644344] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3369.956193] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 3378.721753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3378.774212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3378.827301] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3378.857600] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3378.861537] Lustre: Skipped 1 previous similar message [ 3378.922672] Lustre: lustre-MDT0001: new disk, initializing [ 3378.969867] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3379.003238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3379.017096] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3382.910190] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 3391.452252] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3396.406756] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3403.335976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3403.409659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3403.415350] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3403.635326] Lustre: lustre-OST0000: new disk, initializing [ 3403.641805] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3403.745467] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3408.900825] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 3409.477302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3409.485360] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3419.372867] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3424.816480] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3425.148984] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3429.363724] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3429.515231] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3435.492663] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3435.498329] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3435.499415] LustreError: Skipped 1 previous similar message [ 3435.503470] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3435.516756] Lustre: Skipped 1 previous similar message [ 3437.316968] Lustre: server umount lustre-OST0000 complete [ 3445.728390] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3445.733113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3445.735734] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3445.737672] Lustre: Skipped 1 previous similar message [ 3445.746372] Lustre: Skipped 1 previous similar message [ 3447.010452] Lustre: server umount lustre-MDT0000 complete [ 3450.858393] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3450.880113] LustreError: Skipped 1 previous similar message [ 3451.199774] LustreError: 115992:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776203389 with bad export cookie 10384626307227990477 [ 3451.205430] LustreError: 166-1: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3451.208579] LustreError: 115992:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3451.443834] Lustre: server umount lustre-MDT0001 complete [ 3460.488339] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 17:49:57 (1776203397) [ 3464.218213] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3480.447517] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [ 3489.551808] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 3500.464392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3508.783810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3516.010316] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3523.372400] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3531.743793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3531.834817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3532.113281] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3532.156434] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3532.291612] Lustre: lustre-MDT0000: new disk, initializing [ 3532.379982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3532.401731] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3537.689895] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 3548.677821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3548.773884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3548.845723] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3548.912568] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3548.922697] Lustre: Skipped 1 previous similar message [ 3549.029647] Lustre: lustre-MDT0001: new disk, initializing [ 3549.141465] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3549.191155] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3549.218583] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3553.933836] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 3564.581449] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3570.433512] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3576.269429] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3576.318931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3576.325789] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3576.533475] Lustre: lustre-OST0000: new disk, initializing [ 3576.540270] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3576.650508] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3580.728283] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 3582.544374] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3582.550137] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3589.403698] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3593.707530] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3593.968220] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3599.034985] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3599.282510] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3603.436464] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3603.437439] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3603.450589] LustreError: Skipped 1 previous similar message [ 3603.451588] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3603.451593] Lustre: Skipped 1 previous similar message [ 3603.515579] Lustre: Skipped 1 previous similar message [ 3607.545314] Lustre: server umount lustre-OST0000 complete [ 3613.667730] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3613.684731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3613.691177] Lustre: Skipped 1 previous similar message [ 3618.784815] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3618.801138] LustreError: Skipped 1 previous similar message [ 3619.815237] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3619.833452] LustreError: Skipped 1 previous similar message [ 3621.201366] LustreError: 121040:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776203559 with bad export cookie 10384626307227991331 [ 3621.203034] LustreError: 166-1: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3621.215312] LustreError: 121040:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3629.907711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3630.246726] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3630.307765] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3634.102158] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 3635.692874] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3642.637586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3646.920436] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 3655.528674] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3663.350577] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3670.603754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3675.649450] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 3683.257501] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3689.357717] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3694.561790] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3694.565702] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3694.578625] Lustre: Skipped 1 previous similar message [ 3694.585494] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3699.337747] Lustre: server umount lustre-OST0000 complete [ 3699.340645] Lustre: Skipped 2 previous similar messages [ 3704.288336] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3704.292336] LustreError: Skipped 1 previous similar message [ 3704.299432] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3704.316066] Lustre: Skipped 1 previous similar message [ 3704.320303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3704.322962] Lustre: Skipped 2 previous similar messages [ 3717.087205] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3721.187467] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3721.219107] LustreError: Skipped 2 previous similar messages [ 3721.359959] LustreError: 125383:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776203659 with bad export cookie 10384626307227992185 [ 3721.372801] LustreError: 166-1: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3721.375082] LustreError: 125383:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3738.705879] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 3742.724240] Key type lgssc unregistered [ 3743.186177] LNet: 129170:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3743.202210] LNet: Removed LNI 192.168.202.154@tcp [ 3744.452267] Key type .llcrypt unregistered [ 3744.454154] Key type ._llcrypt unregistered [ 3764.863906] alg: No test for adler32 (adler32-zlib) [ 3765.618181] Key type ._llcrypt registered [ 3765.620254] Key type .llcrypt registered [ 3765.725207] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [ 3776.104484] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 3777.246717] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3777.516436] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 3777.781587] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 3777.791163] LNet: Accept secure, port 988 [ 3779.503540] Key type lgssc registered [ 3780.794443] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3789.920170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3796.653409] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3802.422470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3808.315701] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3815.170926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3815.229566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3816.430619] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3816.454816] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3816.540597] Lustre: lustre-MDT0000: new disk, initializing [ 3816.631765] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3816.670147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3821.007520] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 3832.051511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3832.144352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3832.330754] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3832.404424] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3832.409331] Lustre: Skipped 1 previous similar message [ 3832.522451] Lustre: lustre-MDT0001: new disk, initializing [ 3832.625520] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3832.667451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3832.693891] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3837.784610] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 3847.044792] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3852.880830] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3859.923616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3860.005481] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3860.010229] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3860.217608] Lustre: lustre-OST0000: new disk, initializing [ 3860.221279] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3860.304617] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3865.542870] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 3866.673740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3866.696489] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3874.574852] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3879.571193] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3879.783753] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3885.473218] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3885.827507] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3892.712721] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3892.721672] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3892.722326] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3892.726048] Lustre: Skipped 1 previous similar message [ 3892.753537] LustreError: Skipped 1 previous similar message [ 3893.890176] Lustre: server umount lustre-OST0000 complete [ 3899.362981] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3899.372706] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3899.413446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3899.425118] Lustre: Skipped 1 previous similar message [ 3902.948871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3902.949634] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3904.267190] Lustre: server umount lustre-MDT0000 complete [ 3907.961223] LustreError: 132446:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776203846 with bad export cookie 1994334965629413793 [ 3907.966766] LustreError: 166-1: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3907.971861] LustreError: 132446:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3908.302161] Lustre: server umount lustre-MDT0001 complete [ 3916.217953] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 17:57:33 (1776203853) [ 3922.741937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3923.101585] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3923.231700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3926.976336] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 3928.547551] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3933.670787] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3934.884211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3935.279546] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3939.502687] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 3948.114369] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3955.090972] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3958.752595] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3958.770156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3958.781270] Lustre: Skipped 1 previous similar message [ 3958.796805] Lustre: Skipped 2 previous similar messages [ 3963.880784] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3964.124494] Lustre: server umount lustre-MDT0000 complete [ 3968.448799] LustreError: 136986:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776203906 with bad export cookie 1994334965629414640 [ 3968.474207] LustreError: 166-1: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3968.974494] Lustre: server umount lustre-MDT0001 complete [ 3976.582670] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 17:58:34 (1776203914) [ 3982.643977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3999.264586] LustreError: 139526:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.154@tcp: failed processing log, type 1: rc = -5 [ 4010.528276] LustreError: 139526:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.154@tcp: failed processing log, type 4: rc = -110 [ 4040.351702] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4046.090370] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 4054.917443] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4057.326774] Lustre: server umount lustre-OST0000 complete [ 4067.112205] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 18:00:04 (1776204004) [ 4073.350418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4073.927418] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4073.955339] LustreError: Skipped 3 previous similar messages [ 4074.038725] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4078.873712] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 4084.202029] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4084.213431] LustreError: Skipped 2 previous similar messages [ 4087.862171] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4093.392941] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 4102.868120] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4109.843351] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4117.206173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4117.677442] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4117.683755] Lustre: Skipped 1 previous similar message [ 4122.742429] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 4132.401186] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4140.520178] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4140.538540] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4140.539673] Lustre: Skipped 2 previous similar messages [ 4140.553232] Lustre: Skipped 2 previous similar messages [ 4145.923742] Lustre: server umount lustre-MDT0000 complete [ 4150.136962] LustreError: 140912:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776204088 with bad export cookie 1994334965629415312 [ 4150.141434] LustreError: 140912:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4150.149430] LustreError: 166-1: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4160.479161] Lustre: 143994:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776204092/real 1776204092] req@00000000d798a65f x1862484586705792/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776204098 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4170.739334] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 18:01:47 (1776204107) [ 4177.852367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4178.241597] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4182.475378] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 4190.860381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4191.148975] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4191.151852] Lustre: Skipped 1 previous similar message [ 4195.540597] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 4202.550834] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4209.088950] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4215.806422] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4220.956867] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 4228.600201] Lustre: lustre-OST0000: deleting orphan objects from 0x0:36 to 0x0:65 [ 4229.462239] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4235.085929] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 4235.292784] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4240.408337] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 4240.764982] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4243.943699] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4243.953311] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4243.946624] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4243.961321] Lustre: Skipped 1 previous similar message [ 4243.983618] Lustre: Skipped 4 previous similar messages [ 4249.510132] Lustre: server umount lustre-OST0000 complete [ 4249.513105] Lustre: Skipped 2 previous similar messages [ 4264.416786] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4264.438235] LustreError: Skipped 3 previous similar messages [ 4265.578570] LustreError: 148150:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776204203 with bad export cookie 1994334965629416383 [ 4265.588777] LustreError: 166-1: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4265.596460] LustreError: 148150:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4274.243594] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 18:03:31 (1776204211) [ 4280.437981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4297.184506] LustreError: 148995:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.154@tcp: failed processing log, type 1: rc = -5 [ 4308.447831] LustreError: 148995:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.154@tcp: failed processing log, type 4: rc = -110 [ 4338.338233] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4338.346705] Lustre: Skipped 1 previous similar message [ 4342.484228] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 4350.017899] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4355.898173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4357.040684] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4357.213753] Lustre: lustre-OST0000: deleting orphan objects from 0x0:36 to 0x0:97 [ 4362.125978] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 4370.766323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4375.186332] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 4383.180475] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4388.623478] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4392.682567] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 4392.869122] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4398.057133] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 4398.282416] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4401.632497] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4401.636026] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4401.638494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4401.638503] Lustre: Skipped 6 previous similar messages [ 4401.661657] Lustre: Skipped 4 previous similar messages [ 4406.351232] Lustre: server umount lustre-MDT0000 complete [ 4406.353721] Lustre: Skipped 2 previous similar messages [ 4411.408694] LustreError: 149003:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776204349 with bad export cookie 1994334965629417412 [ 4411.441556] LustreError: 166-1: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4423.135363] Lustre: 152713:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776204355/real 1776204355] req@0000000060a71e4b x1862484586737472/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776204361 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4432.243591] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 18:06:09 (1776204369) [ 4438.665950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4455.263547] LustreError: 153358:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.154@tcp: failed processing log, type 1: rc = -5 [ 4466.527630] LustreError: 153358:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.154@tcp: failed processing log, type 4: rc = -110 [ 4500.663940] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 4510.049582] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4516.381726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4539.529889] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4539.549086] LustreError: Skipped 9 previous similar messages [ 4539.712057] Lustre: lustre-OST0000: deleting orphan objects from 0x0:36 to 0x0:129 [ 4545.402399] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 4554.274289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4559.040222] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 4566.863894] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4573.366910] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4580.705581] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4580.814497] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4580.823726] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 4581.005371] Lustre: lustre-OST0001: new disk, initializing [ 4581.012413] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 4586.378766] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 4588.578242] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4588.590044] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4596.601838] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4601.322387] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 4601.465478] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4606.076242] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 4606.273690] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4608.993396] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4609.002383] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4609.027500] Lustre: Skipped 1 previous similar message [ 4609.066377] Lustre: Skipped 3 previous similar messages [ 4614.786934] Lustre: server umount lustre-OST0000 complete [ 4614.798473] Lustre: Skipped 2 previous similar messages [ 4619.237804] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4643.295162] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 4646.942348] LustreError: 156304:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776204585 with bad export cookie 1994334965629418357 [ 4646.945525] LustreError: 166-1: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4646.949836] LustreError: 156304:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4652.727043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4656.517297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4660.418859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4664.151471] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4670.267748] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 18:10:07 (1776204607) [ 4671.521465] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4672.765871] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4674.223657] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 18:10:11 (1776204611) [ 4679.816869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4680.014510] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4680.053514] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4680.219779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4680.227410] Lustre: Skipped 6 previous similar messages [ 4684.244628] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 4692.242381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4692.342547] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4692.381219] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4696.944849] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 4704.594524] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4709.763690] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4716.442369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4716.677911] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4722.062576] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 4724.285362] Lustre: lustre-OST0000: deleting orphan objects from 0x0:36 to 0x0:161 [ 4730.273509] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4735.886543] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 4736.151888] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4741.238449] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 4741.493342] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4744.674287] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4744.676785] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4744.676903] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4744.693343] Lustre: Skipped 5 previous similar messages [ 4744.748706] Lustre: Skipped 13 previous similar messages [ 4770.924266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4775.874378] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 4776.424514] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4776.437768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4776.441447] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to (at 0@lo) [ 4776.456404] LustreError: 130676:0:(client.c:1279:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000007a0bb982 x1862484586782272/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/224 e 0 to 0 dl 1776204689 ref 1 fl Rpc:RSQU/0/0 rc -107/-107 job:'' [ 4776.457504] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:162 to 0x280000400:193 [ 4776.485606] Lustre: lustre-OST0000: deleting orphan objects from 0x0:36 to 0x0:193 [ 4784.094445] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4791.150899] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 4791.339829] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4796.487950] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 4796.889613] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4802.029065] Lustre: lustre-MDT0001: haven't heard from client a5e89462-adbe-4a6a-95fe-eb426b010100 (at 192.168.202.54@tcp) in 50 seconds. I think it's dead, and I am evicting it. exp 0000000079264dd7, cur 1776204740 expire 1776204710 last 1776204690 [ 4804.937198] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9f3947b21800.ost_server_uuid 40 [ 4808.118029] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9f3947b21800.ost_server_uuid in IDLE state after 1 sec [ 4817.377193] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4817.387945] LustreError: Skipped 1 previous similar message [ 4832.736690] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4832.752411] LustreError: Skipped 11 previous similar messages [ 4833.802164] LustreError: 159935:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776204771 with bad export cookie 1994334965629419512 [ 4833.806470] LustreError: 166-1: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4833.807208] LustreError: 159935:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4844.824452] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 4847.320114] Key type lgssc unregistered [ 4847.547943] LNet: 166338:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4847.558727] LNet: Removed LNI 192.168.202.154@tcp [ 4848.374386] Key type .llcrypt unregistered [ 4848.378434] Key type ._llcrypt unregistered [ 4872.237959] alg: No test for adler32 (adler32-zlib) [ 4872.996449] Key type ._llcrypt registered [ 4872.998062] Key type .llcrypt registered [ 4873.124427] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 4874.659704] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4875.822351] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 4876.054559] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 4876.057908] LNet: Accept secure, port 988 [ 4877.727155] Key type lgssc registered [ 4878.654825] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4886.858527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4888.160214] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4888.191791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4891.978930] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 4893.671433] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4898.784294] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4900.994389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4901.477952] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4906.071723] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 4915.558648] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4922.616416] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4930.863488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4931.637569] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4936.647575] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 4938.799117] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:225 [ 4944.468538] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4952.983441] Lustre: Failing over lustre-MDT0000 [ 4953.115243] Lustre: server umount lustre-MDT0000 complete [ 4956.128991] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4956.144662] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4956.165687] LustreError: Skipped 2 previous similar messages [ 4961.255279] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4961.274865] LustreError: Skipped 2 previous similar messages [ 4963.295336] Lustre: 167136:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776204894/real 1776204894] req@00000000ce96d451 x1862485739058752/t0(0) o400->MGC192.168.202.154@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776204901 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4963.332154] LustreError: 166-1: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4969.441249] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4969.448604] LustreError: Skipped 8 previous similar messages [ 4971.365132] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4980.718052] Lustre: Evicted from MGS (at 192.168.202.154@tcp) after server handle changed from 0x51bbfb572b8b8553 to 0x51bbfb572b8b8935 [ 4980.742189] Lustre: MGC192.168.202.154@tcp: Connection restored to 192.168.202.154@tcp (at 0@lo) [ 4980.953234] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4981.004316] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4984.675151] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 4986.340126] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4986.353499] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 4986.536250] Lustre: lustre-MDT0000: Denying connection for new client e3e8ae8e-7d27-45ae-9f76-691509a2eb2f (at 192.168.202.54@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:05 [ 4996.580965] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4996.590119] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4996.592806] Lustre: Skipped 3 previous similar messages [ 4996.601921] Lustre: Skipped 1 previous similar message [ 5000.457544] Lustre: server umount lustre-OST0000 complete [ 5003.952667] LustreError: 171674:0:(ldlm_lib.c:2882:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5003.963443] Lustre: 170972:0:(ldlm_lib.c:2288:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5003.972684] Lustre: lustre-MDT0000: Recovery over after 0:17, of 2 clients 0 recovered and 2 were evicted. [ 5004.003120] Lustre: 170972:0:(mdt_handler.c:7535:mdt_postrecov()) lustre-MDT0000: auto trigger paused LFSCK failed: rc = -6 [ 5006.823763] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5006.852387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5010.207199] Lustre: server umount lustre-MDT0000 complete [ 5011.940596] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5011.962875] LustreError: Skipped 9 previous similar messages [ 5014.027054] LustreError: 167801:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776204952 with bad export cookie 5889577289528346933 [ 5014.042288] LustreError: 166-1: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5014.048273] LustreError: 167132:0:(import.c:692:ptlrpc_connect_import_locked()) can't connect to a closed import [ 5020.552703] Lustre: server umount lustre-MDT0001 complete [ 5031.522186] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 5034.208400] Key type lgssc unregistered [ 5034.450668] LNet: 172479:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5034.468617] LNet: Removed LNI 192.168.202.154@tcp [ 5035.347821] Key type .llcrypt unregistered [ 5035.350063] Key type ._llcrypt unregistered [ 5060.301809] alg: No test for adler32 (adler32-zlib) [ 5061.058649] Key type ._llcrypt registered [ 5061.061232] Key type .llcrypt registered [ 5061.160502] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 5062.305281] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5062.731408] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 5062.909759] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 5062.913325] LNet: Accept secure, port 988 [ 5064.583126] Key type lgssc registered [ 5065.504933] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5071.969523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5073.303760] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5073.346245] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5077.574124] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 5078.500371] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5083.617935] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5086.343567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5086.690415] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5090.496364] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 5097.226521] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5103.362714] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5109.726394] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5110.054260] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5114.080171] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 5121.379707] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5123.373290] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:257 [ 5131.753485] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5131.761077] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5131.772161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5133.796553] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5133.803542] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5133.813231] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5136.324207] Lustre: server umount lustre-OST0000 complete [ 5143.016673] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5143.024602] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5143.033576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5143.036927] Lustre: Skipped 1 previous similar message [ 5145.586155] Lustre: server umount lustre-MDT0000 complete [ 5149.024683] LustreError: 173947:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776205087 with bad export cookie 9689539377128384037 [ 5149.035644] LustreError: 166-1: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5149.040653] LustreError: 173947:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5149.353593] Lustre: server umount lustre-MDT0001 complete [ 5160.028483] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 5162.243932] Key type lgssc unregistered [ 5162.484868] LNet: 177620:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5162.490224] LNet: Removed LNI 192.168.202.154@tcp [ 5163.111636] Key type .llcrypt unregistered [ 5163.114921] Key type ._llcrypt unregistered [ 5190.294313] alg: No test for adler32 (adler32-zlib) [ 5191.045327] Key type ._llcrypt registered [ 5191.046879] Key type .llcrypt registered [ 5191.110686] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 5191.995777] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5192.297609] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 5192.435256] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 5192.438434] LNet: Accept secure, port 988 [ 5194.071452] Key type lgssc registered [ 5194.867429] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5200.228809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5201.526693] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5201.581414] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5205.276900] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 5207.009410] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5212.135786] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5212.433463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5212.777736] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5216.545181] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 5224.289669] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5231.278274] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5240.110616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5240.356103] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5244.530907] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 5252.101175] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5253.688324] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:289 [ 5263.844569] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5263.848770] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5263.859811] LustreError: Skipped 1 previous similar message [ 5263.870548] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5263.878235] Lustre: Skipped 1 previous similar message [ 5263.898607] Lustre: Skipped 1 previous similar message [ 5268.781951] Lustre: server umount lustre-OST0000 complete [ 5273.034263] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5273.045707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5273.052341] Lustre: Skipped 1 previous similar message [ 5278.181314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5278.195455] Lustre: Skipped 1 previous similar message [ 5280.225942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5280.233180] Lustre: Skipped 1 previous similar message [ 5285.345254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5285.347901] Lustre: Skipped 1 previous similar message [ 5287.391162] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 5287.527658] Lustre: server umount lustre-MDT0000 complete [ 5290.465984] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5290.484428] LustreError: Skipped 1 previous similar message [ 5291.234669] LustreError: 179350:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776205229 with bad export cookie 13786756420622012833 [ 5291.240932] LustreError: 166-1: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5291.243554] LustreError: 179350:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5291.493639] Lustre: server umount lustre-MDT0001 complete [ 5305.069659] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 5307.713331] Key type lgssc unregistered [ 5307.948562] LNet: 183029:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5307.953044] LNet: Removed LNI 192.168.202.154@tcp [ 5308.707775] Key type .llcrypt unregistered [ 5308.710567] Key type ._llcrypt unregistered [ 5330.708817] alg: No test for adler32 (adler32-zlib) [ 5331.465673] Key type ._llcrypt registered [ 5331.467270] Key type .llcrypt registered [ 5331.622447] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 5332.787523] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5333.050344] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 5333.233098] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 5333.237357] LNet: Accept secure, port 988 [ 5334.959363] Key type lgssc registered [ 5335.738716] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5347.086685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5348.355614] Lustre: *** cfs_fail_loc=135, val=0*** [ 5348.419192] LustreError: 184625:0:(obd_config.c:774:class_setup()) setup lustre-MDT0000 failed (-2) [ 5348.422440] LustreError: 184625:0:(obd_config.c:1999:class_config_llog_handler()) MGC192.168.202.154@tcp: cfg command failed: rc = -2 [ 5348.429492] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5348.429492] [ 5348.441846] LustreError: 15c-8: MGC192.168.202.154@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5348.454991] LustreError: 184586:0:(obd_mount_server.c:1425:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5348.471525] LustreError: 184586:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -2 [ 5348.483434] LustreError: 184586:0:(obd_config.c:827:class_cleanup()) Device 5 not setup [ 5348.492918] LustreError: 184593:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776205286 with bad export cookie 2511096818170475977 [ 5353.952813] Lustre: 184586:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776205286/real 1776205286] req@0000000060339367 x1862486218244928/t0(0) o251->MGC192.168.202.154@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776205292 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 5354.224941] Lustre: server umount lustre-MDT0000 complete [ 5354.232534] LustreError: 184586:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5360.969323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5361.375410] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5361.472774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5366.041318] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 5366.756323] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5371.874969] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5373.708651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5374.004076] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5378.180890] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 5385.971814] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5392.176373] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5394.401496] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5394.405754] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5394.417223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5396.448339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5396.456932] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5400.432742] Lustre: server umount lustre-MDT0000 complete [ 5401.596348] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5401.635555] LustreError: Skipped 1 previous similar message [ 5404.974827] LustreError: 184980:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776205343 with bad export cookie 2511096818170476124 [ 5404.981824] LustreError: 184980:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5404.988810] LustreError: 166-1: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5405.252494] Lustre: server umount lustre-MDT0001 complete [ 5417.249364] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 5419.782243] Key type lgssc unregistered [ 5420.030884] LNet: 187483:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5420.037369] LNet: Removed LNI 192.168.202.154@tcp [ 5420.889649] Key type .llcrypt unregistered [ 5420.894157] Key type ._llcrypt unregistered [ 5444.739250] alg: No test for adler32 (adler32-zlib) [ 5445.499363] Key type ._llcrypt registered [ 5445.503900] Key type .llcrypt registered [ 5445.624620] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 5445.830816] Key type .llcrypt unregistered [ 5445.834503] Key type ._llcrypt unregistered [ 5462.119160] alg: No test for adler32 (adler32-zlib) [ 5462.880798] Key type ._llcrypt registered [ 5462.885399] Key type .llcrypt registered [ 5463.015506] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 5464.664773] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5465.247347] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 5465.501731] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 5465.507052] LNet: Accept secure, port 988 [ 5467.215168] Key type lgssc registered [ 5468.347706] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5476.020691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5493.663360] LustreError: 189762:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.154@tcp: failed processing log, type 1: rc = -5 [ 5524.384293] LustreError: 189762:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.154@tcp: failed processing log, type 4: rc = -110 [ 5554.144656] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5558.037645] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 5564.527117] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5570.253747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5577.154285] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5577.318754] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5577.407842] Lustre: lustre-OST0000: deleting orphan objects from 0x0:291 to 0x0:321 [ 5582.304772] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5582.313390] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 5587.430559] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5591.608928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5592.055681] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5596.916661] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 5605.204343] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5612.414719] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5615.721290] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds in log lustre-OST0000 [ 5629.926901] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5629.964235] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5635.041284] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5635.045600] Lustre: Skipped 1 previous similar message [ 5640.163221] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5640.168375] Lustre: Skipped 1 previous similar message [ 5643.231124] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 5643.358731] Lustre: server umount lustre-OST0000 complete [ 5648.353158] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5648.359425] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5648.374110] Lustre: Skipped 1 previous similar message [ 5648.378531] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5648.385016] Lustre: Skipped 1 previous similar message [ 5650.405354] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5655.087571] Lustre: server umount lustre-MDT0000 complete [ 5655.521372] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5655.551246] LustreError: Skipped 1 previous similar message [ 5658.554993] LustreError: 189768:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776205596 with bad export cookie 14553473942042490503 [ 5658.555569] LustreError: 166-1: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5658.562060] LustreError: 189768:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5658.868582] Lustre: server umount lustre-MDT0001 complete [ 5670.656630] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 5673.714670] Key type lgssc unregistered [ 5674.075146] LNet: 193901:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5674.081422] LNet: Removed LNI 192.168.202.154@tcp [ 5675.093908] Key type .llcrypt unregistered [ 5675.096940] Key type ._llcrypt unregistered [ 5700.216655] alg: No test for adler32 (adler32-zlib) [ 5700.975400] Key type ._llcrypt registered [ 5700.977040] Key type .llcrypt registered [ 5701.111709] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 5702.533886] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5703.699196] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 5703.973914] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 5703.977790] LNet: Accept secure, port 988 [ 5705.647137] Key type lgssc registered [ 5706.923257] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5715.468976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5717.078227] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5717.132918] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5722.223845] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 5722.593411] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5727.721945] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5730.407512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5730.619229] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5735.119794] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 5742.398451] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5748.166671] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5755.069704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5755.403276] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5759.230385] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 5766.121218] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5767.795466] Lustre: lustre-OST0000: deleting orphan objects from 0x0:291 to 0x0:353 [ 5774.521302] Lustre: Failing over lustre-MDT0000 [ 5774.635774] Lustre: server umount lustre-MDT0000 complete [ 5776.803967] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5776.865518] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5776.872040] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5778.400450] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5778.428402] Lustre: Skipped 1 previous similar message [ 5781.894029] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5781.917014] LustreError: Skipped 4 previous similar messages [ 5785.570167] Lustre: 194693:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776205716/real 1776205716] req@0000000030e299b3 x1862486607279040/t0(0) o400->MGC192.168.202.154@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776205723 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 5785.601453] LustreError: 166-1: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5790.093562] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5790.113458] LustreError: Skipped 8 previous similar messages [ 5794.373166] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5801.972441] Lustre: Evicted from MGS (at 192.168.202.154@tcp) after server handle changed from 0x4c297e993878c3d2 to 0x4c297e993878c7ad [ 5802.004976] Lustre: MGC192.168.202.154@tcp: Connection restored to 192.168.202.154@tcp (at 0@lo) [ 5802.201280] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5802.249099] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5806.469735] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5807.111318] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 5807.594748] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 5807.631168] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 5807.669350] Lustre: lustre-OST0000: deleting orphan objects from 0x0:291 to 0x0:385 [ 5812.137530] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire in log lustre-MDT0000 [ 5822.409136] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight in log lustre-client [ 5838.309196] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5838.318373] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5838.329603] Lustre: Skipped 1 previous similar message [ 5838.351391] Lustre: Skipped 1 previous similar message [ 5843.430551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5843.447036] Lustre: Skipped 1 previous similar message [ 5843.761478] Lustre: server umount lustre-OST0000 complete [ 5848.546280] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5848.552267] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5848.562540] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5848.575539] Lustre: Skipped 1 previous similar message [ 5853.603291] Lustre: server umount lustre-MDT0000 complete [ 5853.666547] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5853.681425] LustreError: Skipped 15 previous similar messages [ 5857.362120] LustreError: 195363:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776205795 with bad export cookie 5488056817470195629 [ 5857.366552] LustreError: 166-1: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5857.367904] LustreError: 195363:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5857.645819] Lustre: server umount lustre-MDT0001 complete [ 5871.643795] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 5875.056674] Key type lgssc unregistered [ 5875.514240] LNet: 200520:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5875.537552] LNet: Removed LNI 192.168.202.154@tcp [ 5876.430195] Key type .llcrypt unregistered [ 5876.437982] Key type ._llcrypt unregistered [ 5899.995883] alg: No test for adler32 (adler32-zlib) [ 5900.751648] Key type ._llcrypt registered [ 5900.765145] Key type .llcrypt registered [ 5900.901328] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 5902.168699] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5902.583985] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 5902.713989] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 5902.716400] LNet: Accept secure, port 988 [ 5904.341124] Key type lgssc registered [ 5905.365750] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5912.910594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5914.335420] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5914.391715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5918.494481] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 5919.713337] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5924.833970] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5926.569747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5926.977622] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5930.684496] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 5937.707868] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5942.993689] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5948.176515] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5948.482987] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5952.457638] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 5959.624477] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5964.861723] Lustre: lustre-OST0000: deleting orphan objects from 0x0:387 to 0x0:417 [ 5966.885957] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 5967.921817] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 5977.547812] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 5988.320133] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5988.327610] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5988.339653] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5990.373937] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5990.377939] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5990.392740] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5991.846314] Lustre: server umount lustre-OST0000 complete [ 5995.489402] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5995.502463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5995.508158] Lustre: Skipped 1 previous similar message [ 5995.528979] Lustre: Skipped 2 previous similar messages [ 6000.608582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6000.617112] Lustre: Skipped 1 previous similar message [ 6001.388728] Lustre: server umount lustre-MDT0000 complete [ 6004.542564] LustreError: 201965:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776205942 with bad export cookie 15522563045594147579 [ 6004.547955] LustreError: 201965:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6004.549361] LustreError: 166-1: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6004.834873] Lustre: server umount lustre-MDT0001 complete [ 6015.549845] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 6018.133286] Key type lgssc unregistered [ 6018.419510] LNet: 205804:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6018.441943] LNet: Removed LNI 192.168.202.154@tcp [ 6019.325401] Key type .llcrypt unregistered [ 6019.328213] Key type ._llcrypt unregistered [ 6045.187614] alg: No test for adler32 (adler32-zlib) [ 6045.947536] Key type ._llcrypt registered [ 6045.950349] Key type .llcrypt registered [ 6046.116233] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 6047.151088] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6048.293394] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 6048.656358] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 6048.665396] LNet: Accept secure, port 988 [ 6050.423203] Key type lgssc registered [ 6051.968306] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6060.125280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6061.553538] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6061.658862] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6066.660445] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6067.008892] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 6071.777856] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6075.748817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6076.112766] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6079.893311] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 6086.492171] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6092.275371] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6098.746943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6099.164633] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6103.175058] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 6109.717846] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6112.497323] Lustre: lustre-OST0000: deleting orphan objects from 0x0:387 to 0x0:449 [ 6119.389966] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds in log lustre-OST0000 [ 6130.028362] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds in log lustre-OST0000 [ 6140.805591] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub in log lustre-OST0000 [ 6149.494347] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub in log lustre-OST0000 [ 6162.912980] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6162.922875] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6162.943762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6163.937089] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6163.937795] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6163.938793] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6163.946513] Lustre: Skipped 1 previous similar message [ 6165.771547] Lustre: server umount lustre-OST0000 complete [ 6169.056440] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6169.076508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6174.176693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6174.184165] Lustre: Skipped 2 previous similar messages [ 6179.296671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6183.391613] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 6183.584340] Lustre: server umount lustre-MDT0000 complete [ 6184.418166] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6184.429376] LustreError: Skipped 1 previous similar message [ 6186.415535] LustreError: 207270:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776206124 with bad export cookie 3641367696432011853 [ 6186.421205] LustreError: 207270:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6186.422063] LustreError: 166-1: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6186.652358] Lustre: server umount lustre-MDT0001 complete [ 6197.984254] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 6200.609789] Key type lgssc unregistered [ 6200.923432] LNet: 212478:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6200.954403] LNet: Removed LNI 192.168.202.154@tcp [ 6201.781374] Key type .llcrypt unregistered [ 6201.784180] Key type ._llcrypt unregistered [ 6224.385058] alg: No test for adler32 (adler32-zlib) [ 6225.136693] Key type ._llcrypt registered [ 6225.139433] Key type .llcrypt registered [ 6225.227791] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 6226.047511] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6226.710264] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 6226.836933] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 6226.839432] LNet: Accept secure, port 988 [ 6228.455446] Key type lgssc registered [ 6229.364712] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6236.271571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6237.588177] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6237.668079] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6241.320640] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 6242.789287] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6247.925303] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6249.338041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6249.643594] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6253.756688] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 6260.617288] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6265.773166] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6273.870699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6274.239577] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6278.306239] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 6286.149965] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6287.546980] Lustre: lustre-OST0000: deleting orphan objects from 0x0:387 to 0x0:481 [ 6298.823580] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6298.923548] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6299.131118] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6304.156349] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 6315.406636] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6317.925652] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6329.423849] Lustre: Permanently deactivating lustre-OST0001 [ 6329.436689] Lustre: Setting parameter lustre-OST0001-osc.osc.active in log lustre-client [ 6339.695844] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6353.875352] Lustre: Permanently reactivating lustre-OST0001 [ 6353.895330] Lustre: Modifying parameter lustre-OST0001-osc.osc.active in log lustre-client [ 6353.904909] Lustre: Skipped 2 previous similar messages [ 6360.148990] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6360.189862] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6360.202698] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6360.231290] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to (at 0@lo) [ 6360.318214] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6360.350742] Lustre: lustre-OST0001: deleting orphan objects from 0x0:34 to 0x0:65 [ 6365.664063] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6365.666399] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6365.674145] LustreError: Skipped 1 previous similar message [ 6365.675782] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6365.688302] Lustre: Skipped 2 previous similar messages [ 6365.718524] Lustre: Skipped 1 previous similar message [ 6368.743584] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6369.861075] Lustre: server umount lustre-OST0001 complete [ 6373.863119] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6373.890744] LustreError: Skipped 1 previous similar message [ 6374.883189] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6374.910193] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6374.936822] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6374.967503] Lustre: Skipped 1 previous similar message [ 6376.930520] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6376.944227] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6376.957639] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6378.979172] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6378.993306] LustreError: Skipped 1 previous similar message [ 6380.819683] Lustre: server umount lustre-OST0000 complete [ 6387.684509] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6387.688084] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6387.721828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6387.728697] Lustre: Skipped 2 previous similar messages [ 6390.666638] Lustre: server umount lustre-MDT0000 complete [ 6394.258225] LustreError: 216847:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776206332 with bad export cookie 6007135313158175151 [ 6394.283716] LustreError: 166-1: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6394.851551] Lustre: server umount lustre-MDT0001 complete [ 6407.686254] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 6410.164209] Key type lgssc unregistered [ 6410.452138] LNet: 219093:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6410.462522] LNet: Removed LNI 192.168.202.154@tcp [ 6411.164674] Key type .llcrypt unregistered [ 6411.166847] Key type ._llcrypt unregistered [ 6448.896422] alg: No test for adler32 (adler32-zlib) [ 6449.650623] Key type ._llcrypt registered [ 6449.653505] Key type .llcrypt registered [ 6449.800126] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [ 6465.464966] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 6466.694186] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6467.021895] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 6467.316319] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 6467.321921] LNet: Accept secure, port 988 [ 6469.023504] Key type lgssc registered [ 6470.246426] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6479.297472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6486.129529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6492.769948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6499.495423] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6506.509950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6506.570660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6507.907465] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6507.929788] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 6508.010351] Lustre: lustre-MDT0000: new disk, initializing [ 6508.080076] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6508.092756] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6512.630709] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 6523.593886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6523.695907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6523.799766] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6523.864613] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 6523.873813] Lustre: Skipped 1 previous similar message [ 6523.964810] Lustre: lustre-MDT0001: new disk, initializing [ 6524.032459] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6524.073206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6524.079434] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6528.308812] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 6538.387900] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6544.185468] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6551.023041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6551.067087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6551.070587] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 6551.258992] Lustre: lustre-OST0000: new disk, initializing [ 6551.262939] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 6551.361236] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6555.172296] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 6558.336756] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6558.342271] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6565.612831] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6570.436984] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 6570.680778] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6575.304565] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 6575.490886] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6579.169152] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6579.173080] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6579.182274] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6579.191266] Lustre: Skipped 1 previous similar message [ 6583.446516] Lustre: server umount lustre-OST0000 complete [ 6589.408489] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6589.413674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6589.420823] Lustre: Skipped 2 previous similar messages [ 6589.427902] Lustre: Skipped 1 previous similar message [ 6593.468631] Lustre: server umount lustre-MDT0000 complete [ 6594.528185] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6594.536271] LustreError: Skipped 1 previous similar message [ 6595.563200] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6595.571687] LustreError: Skipped 1 previous similar message [ 6597.076398] LustreError: 223207:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776206535 with bad export cookie 11917296157218048108 [ 6597.079791] LustreError: 166-1: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6597.089434] LustreError: 223207:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6597.475713] Lustre: server umount lustre-MDT0001 complete [ 6606.579399] Lustre: DEBUG MARKER: == conf-sanity test 30a: Big config llog and permanent parameter deletion ========================================================== 18:42:23 (1776206543) [ 6613.437451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6613.814177] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6613.888289] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6617.847389] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 6619.107487] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6624.238264] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6626.670596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6627.007798] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6631.160979] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 6639.170574] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6645.800771] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6652.259263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6652.638954] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6657.703410] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 6665.024169] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6667.970324] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 6674.863051] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 6685.432157] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 6693.665340] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 6710.573115] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 6710.582705] Lustre: Skipped 1 previous similar message [ 6746.564038] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 6746.566611] Lustre: Skipped 4 previous similar messages [ 6815.546747] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 6815.551658] Lustre: Skipped 8 previous similar messages [ 6838.241064] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6838.251474] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6838.266084] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6838.301433] Lustre: Skipped 1 previous similar message [ 6843.394125] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6843.401411] Lustre: Skipped 1 previous similar message [ 6843.883534] Lustre: server umount lustre-OST0000 complete [ 6848.491745] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6848.493900] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6848.504840] Lustre: Skipped 2 previous similar messages [ 6858.724555] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6858.737853] Lustre: Skipped 4 previous similar messages [ 6862.303138] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 6862.487325] Lustre: server umount lustre-MDT0000 complete [ 6863.853053] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6863.863946] LustreError: Skipped 1 previous similar message [ 6866.236282] LustreError: 227745:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776206804 with bad export cookie 11917296157218048955 [ 6866.237988] LustreError: 166-1: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6866.248304] LustreError: 227745:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6880.912241] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 6885.562531] Key type lgssc unregistered [ 6886.053828] LNet: 232665:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6886.070234] LNet: Removed LNI 192.168.202.154@tcp [ 6887.605649] Key type .llcrypt unregistered [ 6887.608701] Key type ._llcrypt unregistered [ 6911.912293] alg: No test for adler32 (adler32-zlib) [ 6912.673546] Key type ._llcrypt registered [ 6912.675711] Key type .llcrypt registered [ 6912.758813] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 6913.863229] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6914.265777] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 6914.544287] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 6914.553264] LNet: Accept secure, port 988 [ 6916.215212] Key type lgssc registered [ 6917.533098] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6926.266057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6927.673838] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6927.722807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6932.248891] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 6932.969294] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6938.092261] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6941.629517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6942.080380] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6946.817961] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 6953.972484] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6960.182420] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6968.464587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6968.968216] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6976.386748] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_default_debug -1 all 8 [ 6987.867923] Lustre: DEBUG MARKER: oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7008.231264] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7008.237864] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7008.252546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7009.253746] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7009.258960] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7009.268404] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7012.542610] Lustre: server umount lustre-OST0000 complete [ 7018.976812] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7018.983915] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7019.009079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7019.013450] Lustre: Skipped 1 previous similar message [ 7023.036514] Lustre: server umount lustre-MDT0000 complete [ 7024.608944] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7024.650245] LustreError: Skipped 1 previous similar message [ 7028.681372] LustreError: 234110:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776206966 with bad export cookie 7576127922412473291 [ 7028.684203] LustreError: 166-1: MGC192.168.202.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7028.694195] LustreError: 234110:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7029.385402] Lustre: server umount lustre-MDT0001 complete [ 7043.595579] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 7046.417514] Key type lgssc unregistered [ 7046.681930] LNet: 237942:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7046.689039] LNet: Removed LNI 192.168.202.154@tcp [ 7047.282694] Key type .llcrypt unregistered [ 7047.285616] Key type ._llcrypt unregistered [ 7073.591032] alg: No test for adler32 (adler32-zlib) [ 7074.346365] Key type ._llcrypt registered [ 7074.355664] Key type .llcrypt registered [ 7074.493627] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 7074.851689] Key type .llcrypt unregistered [ 7074.853908] Key type ._llcrypt unregistered [ 7103.856789] alg: No test for adler32 (adler32-zlib) [ 7104.612665] Key type ._llcrypt registered [ 7104.614691] Key type .llcrypt registered [ 7104.691931] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing set_hostid [ 7120.884369] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing load_modules_local [ 7122.080331] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7122.571758] Lustre: Lustre: Build Version: 2.15.8_2_g69e1f89 [ 7122.808413] LNet: Added LNI 192.168.202.154@tcp [8/256/0/180] [ 7122.811419] LNet: Accept secure, port 988 [ 7124.482187] Key type lgssc registered [ 7125.719334] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7138.146485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7148.587168] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7156.815746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7166.695618] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7169.230894] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 6935 sec ========== 18:51:45 (1776207105) [ 7212.323820] Lustre: DEBUG MARKER: oleg254-server.virtnet: executing unload_modules_local [ 7216.041983] Key type lgssc unregistered [ 7216.296569] LNet: 244203:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7216.301923] LNet: Removed LNI 192.168.202.154@tcp [ 7217.011690] Key type .llcrypt unregistered [ 7217.015717] Key type ._llcrypt unregistered