[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 538614306 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001016] APIC: Switch to symmetric I/O mode setup [ 0.003382] x2apic enabled [ 0.004013] Switched APIC routing to physical x2apic. [ 0.006025] kvm-guest: setup PV IPIs [ 0.009000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009031] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010021] pid_max: default: 32768 minimum: 301 [ 0.011188] LSM: Security Framework initializing [ 0.012071] Yama: becoming mindful. [ 0.013060] SELinux: Initializing. [ 0.014128] *** VALIDATE selinux *** [ 0.022772] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027614] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029056] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030144] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032143] *** VALIDATE tmpfs *** [ 0.034006] *** VALIDATE proc *** [ 0.035314] *** VALIDATE cgroup *** [ 0.036018] *** VALIDATE cgroup2 *** [ 0.037337] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038183] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039014] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040044] Spectre V2 : User space: Vulnerable [ 0.041023] Speculative Store Bypass: Vulnerable [ 0.044318] debug: unmapping init [mem 0xffffffffb6e59000-0xffffffffb6e60fff] [ 0.046271] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047849] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048035] ... version: 2 [ 0.049020] ... bit width: 48 [ 0.050021] ... generic registers: 4 [ 0.051022] ... value mask: 0000ffffffffffff [ 0.052023] ... max period: 00007fffffffffff [ 0.053025] ... fixed-purpose events: 3 [ 0.054022] ... event mask: 000000070000000f [ 0.055400] rcu: Hierarchical SRCU implementation. [ 0.057779] smp: Bringing up secondary CPUs ... [ 0.058691] x86: Booting SMP configuration: [ 0.059043] .... node #0, CPUs: #1 #2 #3 [ 0.063796] smp: Brought up 1 node, 4 CPUs [ 0.065104] smpboot: Max logical packages: 1 [ 0.066032] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.211526] node 0 deferred pages initialised in 113ms [ 0.215147] devtmpfs: initialized [ 0.216294] x86/mm: Memory block size: 128MB [ 0.218957] gcov: version magic: 0x41383552 [ 0.220330] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.221098] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.222324] pinctrl core: initialized pinctrl subsystem [ 0.223298] [ 0.223947] ************************************************************* [ 0.224020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.225018] ** ** [ 0.226020] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.227018] ** ** [ 0.228020] ** This means that this kernel is built to expose internal ** [ 0.229017] ** IOMMU data structures, which may compromise security on ** [ 0.230017] ** your system. ** [ 0.231019] ** ** [ 0.232018] ** If you see this message and you are not debugging the ** [ 0.233021] ** kernel, report this immediately to your vendor! ** [ 0.234018] ** ** [ 0.235020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.236020] ************************************************************* [ 0.237882] NET: Registered protocol family 16 [ 0.238539] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.239080] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.240082] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.242041] cpuidle: using governor menu [ 0.243954] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.246703] PCI: Using configuration type 1 for base access [ 0.249160] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.259160] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.260064] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.262058] cryptd: max_cpu_qlen set to 1000 [ 0.265283] ACPI: Added _OSI(Module Device) [ 0.266020] ACPI: Added _OSI(Processor Device) [ 0.268020] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.270017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.275386] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.281677] ACPI: Interpreter enabled [ 0.283064] ACPI: PM: (supports S0 S3 S4 S5) [ 0.285015] ACPI: Using IOAPIC for interrupt routing [ 0.287116] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.290396] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.300762] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.303044] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.306023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.310186] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.317528] acpiphp: Slot [2] registered [ 0.341231] acpiphp: Slot [5] registered [ 0.344149] acpiphp: Slot [6] registered [ 0.346123] acpiphp: Slot [7] registered [ 0.347316] acpiphp: Slot [8] registered [ 0.349319] acpiphp: Slot [9] registered [ 0.351114] acpiphp: Slot [10] registered [ 0.352086] acpiphp: Slot [3] registered [ 0.353168] acpiphp: Slot [4] registered [ 0.354099] acpiphp: Slot [11] registered [ 0.356100] acpiphp: Slot [12] registered [ 0.357080] acpiphp: Slot [13] registered [ 0.358166] acpiphp: Slot [14] registered [ 0.359079] acpiphp: Slot [15] registered [ 0.360086] acpiphp: Slot [16] registered [ 0.361079] acpiphp: Slot [17] registered [ 0.363092] acpiphp: Slot [18] registered [ 0.364116] acpiphp: Slot [19] registered [ 0.365141] acpiphp: Slot [20] registered [ 0.367126] acpiphp: Slot [21] registered [ 0.368112] acpiphp: Slot [22] registered [ 0.369143] acpiphp: Slot [23] registered [ 0.371215] acpiphp: Slot [24] registered [ 0.373304] acpiphp: Slot [25] registered [ 0.375138] acpiphp: Slot [26] registered [ 0.377300] acpiphp: Slot [27] registered [ 0.379132] acpiphp: Slot [28] registered [ 0.380143] acpiphp: Slot [29] registered [ 0.382478] acpiphp: Slot [30] registered [ 0.383128] acpiphp: Slot [31] registered [ 0.385522] PCI host bridge to bus 0000:00 [ 0.386027] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.389035] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.392034] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.394043] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.396033] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.399034] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.401212] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.404176] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.407495] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.416830] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.421050] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.424020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.425016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.427019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.429443] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.433872] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.436048] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.439093] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.444024] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.457020] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.463023] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.470149] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.475017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.480015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.494014] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.507910] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.518025] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.532022] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.562019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.577044] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.585022] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.594024] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.623016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.635193] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.651021] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.667022] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.701018] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.714418] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.727026] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.736024] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.754024] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.773141] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.784024] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.791016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.810018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.821683] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.824540] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.826509] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.829512] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.831349] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.836034] iommu: Default domain type: Passthrough [ 0.837512] SCSI subsystem initialized [ 0.839166] ACPI: bus type USB registered [ 0.841141] usbcore: registered new interface driver usbfs [ 0.843147] usbcore: registered new interface driver hub [ 0.845641] usbcore: registered new device driver usb [ 0.847198] pps_core: LinuxPPS API ver. 1 registered [ 0.848014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.852080] PTP clock support registered [ 0.854130] EDAC MC: Ver: 3.0.0 [ 0.857216] PCI: Using ACPI for IRQ routing [ 0.859046] NetLabel: Initializing [ 0.860013] NetLabel: domain hash size = 128 [ 0.861016] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.862122] NetLabel: unlabeled traffic allowed by default [ 0.865134] vgaarb: loaded [ 0.867428] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.868014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.877000] clocksource: Switched to clocksource kvm-clock [ 1.005283] VFS: Disk quotas dquot_6.6.0 [ 1.007269] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.009962] *** VALIDATE ramfs *** [ 1.011191] *** VALIDATE hugetlbfs *** [ 1.012782] pnp: PnP ACPI init [ 1.015212] pnp: PnP ACPI: found 6 devices [ 1.032354] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.035607] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.037770] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.039871] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.042303] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.044391] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.047287] NET: Registered protocol family 2 [ 1.049827] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.054849] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.059203] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.064752] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.068486] TCP: Hash tables configured (established 65536 bind 65536) [ 1.071505] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.074430] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.076895] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.078960] NET: Registered protocol family 1 [ 1.080949] RPC: Registered named UNIX socket transport module. [ 1.082564] RPC: Registered udp transport module. [ 1.084407] RPC: Registered tcp transport module. [ 1.086515] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.089365] NET: Registered protocol family 44 [ 1.091984] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.094988] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.097743] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.100688] PCI: CLS 0 bytes, default 64 [ 1.104167] Unpacking initramfs... [ 2.540765] debug: unmapping init [mem 0xffff998e3cc54000-0xffff998e3ffbffff] [ 2.546184] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.549596] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.554301] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.064195] Initialise system trusted keyrings [ 3.066140] Key type blacklist registered [ 3.068468] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.080500] zbud: loaded [ 3.084190] *** VALIDATE nfs *** [ 3.085621] *** VALIDATE nfs4 *** [ 3.087509] pstore: using deflate compression [ 3.091615] Platform Keyring initialized [ 3.222271] NET: Registered protocol family 38 [ 3.224279] Key type asymmetric registered [ 3.225517] Asymmetric key parser 'x509' registered [ 3.227312] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.229983] io scheduler mq-deadline registered [ 3.232261] io scheduler kyber registered [ 3.234229] io scheduler bfq registered [ 3.236245] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.238806] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.241320] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.243768] ACPI: Power Button [PWRF] [ 3.247704] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.252460] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.265057] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.272025] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.298527] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.328259] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.357588] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.363512] Non-volatile memory driver v1.3 [ 3.365843] Linux agpgart interface v0.103 [ 3.404090] virtio_blk virtio1: [vda] 68040 512-byte logical blocks (34.8 MB/33.2 MiB) [ 3.407098] vda: detected capacity change from 0 to 34836480 [ 3.435342] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.438493] vdb: detected capacity change from 0 to 1073741824 [ 3.454736] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.457927] vdc: detected capacity change from 0 to 2621440000 [ 3.471129] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.473251] vdd: detected capacity change from 0 to 2621440000 [ 3.488317] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.491889] vde: detected capacity change from 0 to 4294967296 [ 3.506181] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.508686] vdf: detected capacity change from 0 to 4294967296 [ 3.515468] libphy: Fixed MDIO Bus: probed [ 3.521112] usbcore: registered new interface driver usbserial_generic [ 3.523258] usbserial: USB Serial support registered for generic [ 3.525240] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.529405] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.530747] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.532827] mousedev: PS/2 mouse device common for all mice [ 3.535499] rtc_cmos 00:05: RTC can wake from S4 [ 3.538482] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.541971] rtc_cmos 00:05: registered as rtc0 [ 3.544213] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.544701] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.550073] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.553094] intel_pstate: CPU model not supported [ 3.559064] hid: raw HID events driver (C) Jiri Kosina [ 3.560776] usbcore: registered new interface driver usbhid [ 3.562543] usbhid: USB HID core driver [ 3.563836] drop_monitor: Initializing network drop monitor service [ 3.565898] Initializing XFRM netlink socket [ 3.567515] NET: Registered protocol family 10 [ 3.569788] Segment Routing with IPv6 [ 3.570747] NET: Registered protocol family 17 [ 3.572212] mpls_gso: MPLS GSO support [ 3.579258] RAS: Correctable Errors collector initialized. [ 3.581712] AVX version of gcm_enc/dec engaged. [ 3.583622] AES CTR mode by8 optimization enabled [ 3.664367] sched_clock: Marking stable (3664346038, 0)->(4710848518, -1046502480) [ 3.667153] registered taskstats version 1 [ 3.669167] Loading compiled-in X.509 certificates [ 3.670552] zswap: loaded using pool lzo/zbud [ 3.696989] Key type big_key registered [ 3.712065] Key type encrypted registered [ 3.713705] ima: No TPM chip found, activating TPM-bypass! [ 3.715754] ima: Allocated hash algorithm: sha1 [ 3.717458] ima: No architecture policies found [ 3.719101] evm: Initialising EVM extended attributes: [ 3.720993] evm: security.selinux [ 3.722237] evm: security.ima [ 3.723331] evm: security.capability [ 3.724584] evm: HMAC attrs: 0x1 [ 3.726850] rtc_cmos 00:05: setting system clock to 2026-04-14 18:51:24 UTC (1776192684) [ 3.733517] debug: unmapping init [mem 0xffffffffb7e03000-0xffffffffb7ffffff] [ 3.736290] debug: unmapping init [mem 0xffffffffb6b82000-0xffffffffb6e58fff] [ 3.743138] Write protecting the kernel read-only data: 28672k [ 3.745728] debug: unmapping init [mem 0xffffffffb5203000-0xffffffffb53fffff] [ 3.747756] debug: unmapping init [mem 0xffffffffb5b14000-0xffffffffb5bfffff] [ 3.803481] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.811907] systemd[1]: Detected virtualization kvm. [ 3.813897] systemd[1]: Detected architecture x86-64. [ 3.815816] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.842709] systemd[1]: No hostname configured. [ 3.844549] systemd[1]: Set hostname to . [ 3.846632] random: systemd: uninitialized urandom read (16 bytes read) [ 3.849140] systemd[1]: Initializing machine ID from random generator. [ 3.894224] random: ln: uninitialized urandom read (6 bytes read) [ 4.001906] random: systemd: uninitialized urandom read (16 bytes read) [ 4.004810] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 4.010461] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 4.015341] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. [ OK ] Reached target Swap. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Reached target Paths. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.645535] device-mapper: uevent: version 1.0.3 [ 4.647910] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.297454] random: fast init done [ 5.307966] virtio_net virtio0 ens2: renamed from eth0 [ 5.371051] scsi host0: ata_piix [ 5.383267] scsi host1: ata_piix [ 5.384898] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.387427] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.763515] dracut-initqueue[588]: RTNETLINK answers: File exists [ 10.229347] random: crng init done [ 10.230823] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.636152] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.904957] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.174408] SELinux: Disabled at runtime. [ 12.232251] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.240249] systemd[1]: Detected virtualization kvm. [ 12.242115] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.773370] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.776818] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.781666] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.787496] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.791711] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.801290] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.813352] systemd[1]: Listening on RPCbind Server Activation Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Switch Root. Mounting POSIX Message Queue File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-getty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Stopped target Initrd Root File System. Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. Mounting Kernel Debug File System... [ OK ] Stopped target Initrd File Systems. Mounting Huge Pages File System... Starting Create list of required st…ce nodes for the current kernel... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ 13.017642] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 13.287208] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.590355] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.698214] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.854974] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.880351] EDAC sbridge: Ver: 1.1.2 [ 15.433276] Key type dns_resolver registered [ 15.769665] NFS: Registering the id_resolver key type [ 15.771698] Key type id_resolver registered [ 15.773584] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started RPC Bind. [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg216-server login: [ 56.730047] libcfs: loading out-of-tree module taints kernel. [ 56.839554] alg: No test for adler32 (adler32-zlib) [ 57.590715] Key type ._llcrypt registered [ 57.594387] Key type .llcrypt registered [ 57.786114] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 79.855409] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 81.944680] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 82.919199] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 83.642566] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 83.647223] LNet: Accept secure, port 988 [ 85.456846] Key type lgssc registered [ 87.436241] Lustre: Echo OBD driver; http://www.lustre.org/ [ 87.716150] hrtimer: interrupt took 15641889 ns [ 103.694812] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 109.001154] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 122.427544] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 133.075438] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 142.476787] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 160.753394] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 175.371824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 175.540096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 177.070251] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 177.126302] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 177.278828] Lustre: lustre-MDT0000: new disk, initializing [ 177.427396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 177.461095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 183.359511] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 198.668843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 198.752472] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 198.859647] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 198.891992] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 198.903573] Lustre: Skipped 1 previous similar message [ 199.095994] Lustre: lustre-MDT0001: new disk, initializing [ 199.174517] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 199.245460] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 199.259306] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 204.841559] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 220.432511] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 220.515106] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 220.522577] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 220.876449] Lustre: lustre-OST0000: new disk, initializing [ 220.883564] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 220.980121] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 226.034246] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 226.899061] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 226.911968] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 241.093376] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 241.156523] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 241.161307] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 241.303763] Lustre: lustre-OST0001: new disk, initializing [ 241.307929] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 241.422360] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 245.158119] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 250.978219] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 250.996511] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 256.776106] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 263.517488] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 271.924203] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing check_logdir /tmp/testlogs/ [ 280.180470] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing yml_node [ 285.626517] Lustre: DEBUG MARKER: Client: 2.15.8.10 [ 288.520771] Lustre: DEBUG MARKER: MDS: 2.15.8.10 [ 292.173134] Lustre: DEBUG MARKER: OSS: 2.15.8.10 [ 294.513951] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Apr 14 14:56:12 EDT 2026 [ 304.197385] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball 110 [ 305.994189] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 [ 322.018975] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 322.033479] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 322.067687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 323.041317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 323.046849] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 323.081842] Lustre: Skipped 2 previous similar messages [ 328.163725] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 328.187037] LustreError: Skipped 3 previous similar messages [ 329.959334] Lustre: server umount lustre-MDT0000 complete [ 333.281635] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 333.299417] LustreError: Skipped 3 previous similar messages [ 333.793604] LustreError: 8249:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776193014 with bad export cookie 9833555552242965300 [ 333.801593] LustreError: 166-1: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 333.802334] LustreError: 8249:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 338.400391] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 338.401883] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 338.416165] LustreError: Skipped 1 previous similar message [ 338.470223] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 338.474787] Lustre: Skipped 4 previous similar messages [ 344.545156] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 344.546091] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 344.583730] LustreError: Skipped 2 previous similar messages [ 344.628074] Lustre: Skipped 3 previous similar messages [ 348.127439] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 348.479411] Lustre: server umount lustre-MDT0001 complete [ 353.200555] Lustre: server umount lustre-OST0000 complete [ 357.637668] Lustre: server umount lustre-OST0001 complete [ 365.261394] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 374.246335] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 386.561699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 393.120128] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 398.157662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 403.916573] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 410.595451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 410.684406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 410.967842] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 410.991388] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 411.099306] Lustre: lustre-MDT0000: new disk, initializing [ 411.226053] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 411.252571] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 415.609934] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 426.612413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 426.680553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 426.736409] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 426.764262] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 426.766397] Lustre: Skipped 1 previous similar message [ 426.827375] Lustre: lustre-MDT0001: new disk, initializing [ 426.933387] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 426.994184] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 427.014692] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 430.998635] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 441.351606] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 446.548304] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 454.087817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 454.180385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 454.186272] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 454.581327] Lustre: lustre-OST0000: new disk, initializing [ 454.584662] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 454.658933] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 455.787850] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 455.807241] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 459.140272] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 471.078751] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 477.175528] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 477.393974] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 482.581976] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 482.739383] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 486.884041] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 486.909434] Lustre: Skipped 1 previous similar message [ 486.924901] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 486.925974] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 486.961039] Lustre: Skipped 1 previous similar message [ 497.129891] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 497.141540] Lustre: Skipped 3 previous similar messages [ 499.167918] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 499.394742] Lustre: server umount lustre-OST0000 complete [ 503.790524] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 503.804616] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 503.813447] Lustre: Skipped 1 previous similar message [ 517.600209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 517.608847] Lustre: Skipped 6 previous similar messages [ 518.111142] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 518.446899] Lustre: server umount lustre-MDT0000 complete [ 522.114481] LustreError: 14339:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776193202 with bad export cookie 9833555552242966756 [ 522.116369] LustreError: 166-1: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 522.126013] LustreError: 14339:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 526.437366] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 15:00:05 (1776193205) [ 534.052785] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 534.517521] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 534.535555] LustreError: Skipped 1 previous similar message [ 534.595568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 539.699437] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 544.737559] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 544.765442] LustreError: Skipped 1 previous similar message [ 549.714060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 555.240516] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 562.568735] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 569.556369] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 576.746525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 583.250485] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 593.403103] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 600.034813] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 600.039118] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 600.092184] Lustre: Skipped 1 previous similar message [ 600.112052] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 605.985040] Lustre: server umount lustre-OST0000 complete [ 605.988415] Lustre: Skipped 1 previous similar message [ 611.811955] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 620.512553] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 620.532324] LustreError: Skipped 2 previous similar messages [ 621.105804] LustreError: 18733:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776193301 with bad export cookie 9833555552242967610 [ 621.120092] LustreError: 166-1: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 634.426820] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 637.576754] Key type lgssc unregistered [ 637.961634] LNet: 22423:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 637.970518] LNet: Removed LNI 192.168.202.116@tcp [ 638.946721] Key type .llcrypt unregistered [ 638.949441] Key type ._llcrypt unregistered [ 661.859362] alg: No test for adler32 (adler32-zlib) [ 662.622819] Key type ._llcrypt registered [ 662.630162] Key type .llcrypt registered [ 662.826940] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 664.216401] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 665.297104] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 665.700947] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 665.707843] LNet: Accept secure, port 988 [ 667.375168] Key type lgssc registered [ 668.777775] Lustre: Echo OBD driver; http://www.lustre.org/ [ 676.991846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 678.578102] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 678.632813] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 684.000977] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 684.297048] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 689.128258] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 694.268127] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 696.476536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 696.967817] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 701.703262] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 709.228500] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 715.447773] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 723.264111] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 723.825144] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 729.013912] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 729.975935] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 737.068961] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 750.568970] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 750.603489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 750.603868] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 750.619979] LustreError: Skipped 1 previous similar message [ 752.429891] Lustre: server umount lustre-OST0000 complete [ 758.240553] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 758.250275] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 758.275946] Lustre: Skipped 1 previous similar message [ 758.278654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 758.285568] Lustre: Skipped 1 previous similar message [ 760.800064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 760.805670] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 762.212906] Lustre: server umount lustre-MDT0000 complete [ 765.930129] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 765.941539] LustreError: Skipped 1 previous similar message [ 766.891155] LustreError: 23871:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776193447 with bad export cookie 17647767553781115083 [ 766.904092] LustreError: 166-1: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 766.906520] LustreError: 23871:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 767.317505] Lustre: server umount lustre-MDT0001 complete [ 778.845484] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 781.621621] Key type lgssc unregistered [ 781.843588] LNet: 27894:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 781.853847] LNet: Removed LNI 192.168.202.116@tcp [ 782.538142] Key type .llcrypt unregistered [ 782.540731] Key type ._llcrypt unregistered [ 808.339558] alg: No test for adler32 (adler32-zlib) [ 809.097460] Key type ._llcrypt registered [ 809.101235] Key type .llcrypt registered [ 809.280573] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 810.631875] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 811.464383] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 811.640809] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 811.648749] LNet: Accept secure, port 988 [ 813.295329] Key type lgssc registered [ 814.466728] Lustre: Echo OBD driver; http://www.lustre.org/ [ 822.434654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 823.999328] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 824.092058] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 828.244627] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 829.411038] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 834.533216] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 837.289391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 837.825044] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 842.399551] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 850.527941] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 857.955337] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 874.002590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 874.283536] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 875.338866] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 878.686407] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 884.864986] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 895.977872] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 895.986898] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 896.008701] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 898.023339] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 898.025977] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 898.059749] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 902.180144] Lustre: server umount lustre-OST0000 complete [ 908.258071] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 908.258889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 908.258895] Lustre: Skipped 1 previous similar message [ 908.311321] Lustre: Skipped 1 previous similar message [ 912.864454] Lustre: server umount lustre-MDT0000 complete [ 913.385231] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 918.794593] LustreError: 29361:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776193599 with bad export cookie 2501574761767333540 [ 918.795066] LustreError: 166-1: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 918.813237] LustreError: 29361:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 919.248923] Lustre: server umount lustre-MDT0001 complete [ 934.886378] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 937.196892] Key type lgssc unregistered [ 937.558708] LNet: 33387:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 937.566898] LNet: Removed LNI 192.168.202.116@tcp [ 938.450781] Key type .llcrypt unregistered [ 938.453983] Key type ._llcrypt unregistered [ 960.396542] alg: No test for adler32 (adler32-zlib) [ 961.155422] Key type ._llcrypt registered [ 961.158957] Key type .llcrypt registered [ 961.292797] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 962.518389] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 962.934530] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 963.116023] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 963.120387] LNet: Accept secure, port 988 [ 964.824068] Key type lgssc registered [ 965.980606] Lustre: Echo OBD driver; http://www.lustre.org/ [ 974.750095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 976.151027] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 976.225225] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 981.432308] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 981.473342] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 986.600968] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 990.121497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 990.329503] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 995.690923] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 1004.470944] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1010.503857] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1019.373110] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1020.000602] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1024.489536] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 1027.188440] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 1034.329972] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1047.013877] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1047.033866] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1047.056177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1048.043695] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1048.044995] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1050.009511] Lustre: server umount lustre-OST0000 complete [ 1056.736238] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1056.745764] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1056.758870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1056.774633] Lustre: Skipped 1 previous similar message [ 1059.899526] Lustre: server umount lustre-MDT0000 complete [ 1063.402866] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1064.057706] LustreError: 34829:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776193744 with bad export cookie 9746063972128696990 [ 1064.068982] LustreError: 166-1: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1064.072037] LustreError: 34829:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1064.526124] Lustre: server umount lustre-MDT0001 complete [ 1077.803524] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 1080.853521] Key type lgssc unregistered [ 1081.088635] LNet: 38505:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1081.099244] LNet: Removed LNI 192.168.202.116@tcp [ 1081.837430] Key type .llcrypt unregistered [ 1081.839509] Key type ._llcrypt unregistered [ 1105.378447] alg: No test for adler32 (adler32-zlib) [ 1106.141902] Key type ._llcrypt registered [ 1106.144650] Key type .llcrypt registered [ 1106.241905] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 1107.906053] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1108.394813] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 1108.558106] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 1108.563141] LNet: Accept secure, port 988 [ 1110.255569] Key type lgssc registered [ 1111.528478] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1119.547228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1120.981533] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1121.051979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1125.528113] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 1126.373103] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1131.498408] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1133.808031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1134.200562] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1139.890183] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 1148.618837] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1154.388833] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1160.878230] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1161.206817] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1166.164922] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 1171.447036] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 1174.417852] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1181.082633] Lustre: lustre-OST0000: Not available for connect from 192.168.202.16@tcp (stopping) [ 1181.665353] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1181.673649] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1181.698619] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1181.707878] Lustre: Skipped 1 previous similar message [ 1186.027838] Lustre: server umount lustre-OST0000 complete [ 1205.143682] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1205.155203] Lustre: Skipped 1 previous similar message [ 1205.169119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1205.179290] Lustre: Skipped 1 previous similar message [ 1210.336803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1210.355828] Lustre: Skipped 1 previous similar message [ 1211.541225] Lustre: server umount lustre-MDT0000 complete [ 1213.449596] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1213.471789] LustreError: Skipped 1 previous similar message [ 1215.822207] LustreError: 39966:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776193896 with bad export cookie 4296744051807226270 [ 1215.825198] LustreError: 166-1: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1215.839978] LustreError: 39966:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1230.306225] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1230.639596] Lustre: server umount lustre-MDT0001 complete [ 1242.927603] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 1245.981812] Key type lgssc unregistered [ 1246.228571] LNet: 43818:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1246.244725] LNet: Removed LNI 192.168.202.116@tcp [ 1247.332772] Key type .llcrypt unregistered [ 1247.335945] Key type ._llcrypt unregistered [ 1269.368281] alg: No test for adler32 (adler32-zlib) [ 1270.127409] Key type ._llcrypt registered [ 1270.130118] Key type .llcrypt registered [ 1270.319821] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 1271.569422] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1272.317183] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 1272.490539] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 1272.499710] LNet: Accept secure, port 988 [ 1274.183136] Key type lgssc registered [ 1275.494182] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1283.565312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1285.149740] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1285.242577] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1289.248287] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1291.043475] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 1294.307264] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1299.455458] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1299.913468] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1300.514068] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1305.293664] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 1315.039713] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1322.937620] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1329.426699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1329.881621] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1334.314807] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 1335.033670] Lustre: lustre-OST0000: deleting orphan objects from 0x0:163 to 0x0:193 [ 1341.947555] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1348.995551] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.16@tcp (stopping) [ 1350.632302] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1350.635792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1350.655489] Lustre: Skipped 2 previous similar messages [ 1350.673490] Lustre: Skipped 2 previous similar messages [ 1354.106784] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.16@tcp (stopping) [ 1354.592712] Lustre: server umount lustre-MDT0000 complete [ 1355.745026] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1355.762760] LustreError: Skipped 2 previous similar messages [ 1359.690995] LustreError: 45261:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776194040 with bad export cookie 4667312790820911656 [ 1359.699200] LustreError: 166-1: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1359.702681] LustreError: 45261:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1360.872870] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1360.892804] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1364.341579] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1364.368346] LustreError: Skipped 2 previous similar messages [ 1364.900163] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1364.912183] Lustre: Skipped 1 previous similar message [ 1366.415468] Lustre: server umount lustre-MDT0001 complete [ 1382.075414] Lustre: server umount lustre-OST0000 complete [ 1401.841249] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 1404.430664] Key type lgssc unregistered [ 1404.690925] LNet: 49266:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1404.699499] LNet: Removed LNI 192.168.202.116@tcp [ 1405.394324] Key type .llcrypt unregistered [ 1405.396567] Key type ._llcrypt unregistered [ 1431.777241] alg: No test for adler32 (adler32-zlib) [ 1432.544505] Key type ._llcrypt registered [ 1432.546860] Key type .llcrypt registered [ 1432.691481] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 1433.649277] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1434.520365] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 1434.880427] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 1434.888289] LNet: Accept secure, port 988 [ 1436.623132] Key type lgssc registered [ 1438.402759] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1446.992016] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1464.737672] LustreError: 50718:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.116@tcp: failed processing log, type 1: rc = -5 [ 1495.520839] LustreError: 50718:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.116@tcp: failed processing log, type 4: rc = -110 [ 1526.304821] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1531.722361] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 1539.243289] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1579.228825] Lustre: server umount lustre-OST0000 complete [ 1600.692723] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 1603.438421] Key type lgssc unregistered [ 1603.724542] LNet: 52392:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1603.737907] LNet: Removed LNI 192.168.202.116@tcp [ 1604.606221] Key type .llcrypt unregistered [ 1604.607798] Key type ._llcrypt unregistered [ 1628.384945] alg: No test for adler32 (adler32-zlib) [ 1629.138461] Key type ._llcrypt registered [ 1629.141661] Key type .llcrypt registered [ 1629.266255] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 1630.695659] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1631.438214] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 1631.740248] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 1631.747956] LNet: Accept secure, port 988 [ 1633.439132] Key type lgssc registered [ 1634.859617] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1645.060649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1646.625965] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1646.737765] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1652.194038] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1652.279992] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 1657.316078] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1661.411878] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1662.540681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1662.836219] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1667.250306] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 1676.393405] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1683.278411] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1690.423314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1690.848106] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1697.020824] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:225 [ 1697.442969] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 1704.772713] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1707.490876] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1707.505867] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1712.624417] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1712.633134] Lustre: Skipped 2 previous similar messages [ 1713.415248] Lustre: server umount lustre-OST0000 complete [ 1717.730984] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1717.737391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1717.768184] Lustre: Skipped 2 previous similar messages [ 1717.797102] Lustre: Skipped 1 previous similar message [ 1722.849479] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1722.854671] Lustre: Skipped 1 previous similar message [ 1723.758656] Lustre: server umount lustre-MDT0000 complete [ 1727.757868] LustreError: 53851:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776194408 with bad export cookie 10783524223727762022 [ 1727.781257] LustreError: 166-1: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1728.160370] Lustre: server umount lustre-MDT0001 complete [ 1739.916226] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 1742.270515] Key type lgssc unregistered [ 1742.503818] LNet: 57523:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1742.517565] LNet: Removed LNI 192.168.202.116@tcp [ 1743.152533] Key type .llcrypt unregistered [ 1743.158285] Key type ._llcrypt unregistered [ 1763.758156] alg: No test for adler32 (adler32-zlib) [ 1764.518174] Key type ._llcrypt registered [ 1764.521527] Key type .llcrypt registered [ 1764.652857] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 1766.121631] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1766.779991] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 1766.968917] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 1766.975308] LNet: Accept secure, port 988 [ 1768.622189] Key type lgssc registered [ 1770.107067] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1778.874394] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1796.513951] LustreError: 58961:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.116@tcp: failed processing log, type 1: rc = -5 [ 1827.231284] LustreError: 58961:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.116@tcp: failed processing log, type 4: rc = -110 [ 1858.015370] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1863.221893] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 1871.020961] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1877.248978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1879.863177] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1879.921977] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1879.984810] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:257 [ 1884.786677] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 1885.160194] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1890.272957] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1893.355345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1893.617114] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1898.399338] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 1906.559884] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1912.452091] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1915.874920] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1915.890411] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1915.914220] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1917.408911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1917.410852] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1922.543568] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1922.550245] Lustre: Skipped 2 previous similar messages [ 1927.649908] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1927.660257] Lustre: Skipped 1 previous similar message [ 1929.183192] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1929.345460] Lustre: server umount lustre-OST0000 complete [ 1953.248983] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1953.293313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1957.791880] Lustre: server umount lustre-MDT0000 complete [ 1958.369402] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1958.390327] LustreError: Skipped 1 previous similar message [ 1962.452804] LustreError: 58967:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776194643 with bad export cookie 13034001180714651744 [ 1962.460341] LustreError: 166-1: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1962.473804] LustreError: 58967:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1968.948802] Lustre: server umount lustre-MDT0001 complete [ 1979.731945] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 1982.510377] Key type lgssc unregistered [ 1982.804821] LNet: 62834:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1982.829359] LNet: Removed LNI 192.168.202.116@tcp [ 1983.632507] Key type .llcrypt unregistered [ 1983.650352] Key type ._llcrypt unregistered [ 2009.431748] alg: No test for adler32 (adler32-zlib) [ 2010.194561] Key type ._llcrypt registered [ 2010.197632] Key type .llcrypt registered [ 2010.361109] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 2012.024416] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2012.531333] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 2012.763530] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 2012.771498] LNet: Accept secure, port 988 [ 2014.407231] Key type lgssc registered [ 2015.693834] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2023.944466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2025.401440] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2025.510819] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2029.505796] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 2030.565913] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2035.684310] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2038.447879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2038.783952] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2044.496490] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 2052.672351] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2058.572376] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2065.033138] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2065.390574] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2069.650396] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 2075.644658] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:289 [ 2078.295933] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2096.096864] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2096.099843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2096.115200] Lustre: Skipped 1 previous similar message [ 2096.983457] Lustre: server umount lustre-OST0000 complete [ 2101.216629] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2101.229817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2101.232728] Lustre: Skipped 2 previous similar messages [ 2106.336295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2106.342220] Lustre: Skipped 1 previous similar message [ 2106.555568] Lustre: server umount lustre-MDT0000 complete [ 2109.860779] LustreError: 64299:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776194790 with bad export cookie 17813525576241389636 [ 2109.883761] LustreError: 166-1: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2110.175242] Lustre: server umount lustre-MDT0001 complete [ 2121.931381] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 2124.791535] Key type lgssc unregistered [ 2125.043901] LNet: 67991:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2125.054898] LNet: Removed LNI 192.168.202.116@tcp [ 2125.807586] Key type .llcrypt unregistered [ 2125.811389] Key type ._llcrypt unregistered [ 2155.430694] alg: No test for adler32 (adler32-zlib) [ 2156.193553] Key type ._llcrypt registered [ 2156.200410] Key type .llcrypt registered [ 2156.300471] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 2157.442552] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2157.754218] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 2157.965834] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 2157.971674] LNet: Accept secure, port 988 [ 2159.631132] Key type lgssc registered [ 2160.662319] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2167.627837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2168.993148] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2169.046113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2172.851552] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 2174.445877] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2179.563640] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2181.227469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2181.731308] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2187.023205] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 2195.584745] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2202.544427] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2209.654408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2210.011989] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2214.520952] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 2219.262604] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:321 [ 2221.408318] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2226.846102] Lustre: Failing over lustre-MDT0000 [ 2227.130199] Lustre: server umount lustre-MDT0000 complete [ 2227.686750] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2227.704058] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2227.726529] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2229.739398] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2229.761307] Lustre: Skipped 1 previous similar message [ 2233.199423] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2233.229032] LustreError: Skipped 4 previous similar messages [ 2236.708954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2236.777689] Lustre: 69058:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776194910/real 1776194910] req@00000000413fd31c x1862475278464448/t0(0) o400->MGC192.168.202.116@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776194917 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2236.814834] LustreError: 166-1: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2241.403898] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2241.448749] LustreError: Skipped 8 previous similar messages [ 2243.068650] Lustre: Evicted from MGS (at 192.168.202.116@tcp) after server handle changed from 0xf7cec4a4918df5d3 to 0xf7cec4a4918df9c3 [ 2243.087890] Lustre: MGC192.168.202.116@tcp: Connection restored to (at 0@lo) [ 2243.316864] Lustre: *** cfs_fail_loc=135, val=0*** [ 2243.489819] BUG: spinlock bad magic on CPU#0, llog_process_th/72905 [ 2243.499641] lock: 0xffff998ebe25acc0, .magic: 00000000, .owner: /-1, .owner_cpu: 0 [ 2243.512373] CPU: 0 PID: 72905 Comm: llog_process_th Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 [ 2243.520625] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 2243.530954] Call Trace: [ 2243.535414] ? dump_stack+0xbb/0x10e [ 2243.536869] ? spin_dump.cold.3+0x32/0x65 [ 2243.549841] ? do_raw_spin_lock+0xce/0x190 [ 2243.554024] ? _raw_spin_lock+0x21/0x30 [ 2243.559540] ? mdd_changelog_fini+0x45/0x3a0 [mdd] [ 2243.570337] ? mdd_process_config+0x368/0x9c0 [mdd] [ 2243.576377] ? mdt_stack_fini+0x5de/0x1200 [mdt] [ 2243.586455] ? mdt_init0+0x650/0x1750 [mdt] [ 2243.591601] ? mdt_device_alloc+0x118/0x170 [mdt] [ 2243.596572] ? obd_setup+0x196/0x3e0 [obdclass] [ 2243.603889] ? class_setup+0x6ee/0x9e0 [obdclass] [ 2243.612599] ? class_process_config+0x17c7/0x2b90 [obdclass] [ 2243.624843] ? class_config_llog_handler+0xb73/0x1ab0 [obdclass] [ 2243.633547] ? llog_process_thread+0x138f/0x2270 [obdclass] [ 2243.638222] ? mgs_key_init+0xdd/0x110 [mgs] [ 2243.639724] ? keys_fill+0x10d/0x1b0 [obdclass] [ 2243.645966] ? lu_context_init+0xc8/0x240 [obdclass] [ 2243.652422] ? llog_process+0x40/0x40 [obdclass] [ 2243.657983] ? llog_process_thread_daemonize+0xe1/0x150 [obdclass] [ 2243.661864] ? kthread+0x1d1/0x200 [ 2243.666027] ? set_kthread_struct+0x70/0x70 [ 2243.667572] ? ret_from_fork+0x1f/0x30 [ 2243.675684] LustreError: 72905:0:(obd_config.c:774:class_setup()) setup lustre-MDT0000 failed (-2) [ 2243.690908] LustreError: 72905:0:(obd_config.c:1999:class_config_llog_handler()) MGC192.168.202.116@tcp: cfg command failed: rc = -2 [ 2243.700175] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2243.700175] [ 2243.714871] LustreError: 15c-8: MGC192.168.202.116@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2243.738710] LustreError: 72892:0:(obd_mount_server.c:1425:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2243.764586] LustreError: 72892:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -2 [ 2243.780244] LustreError: 72892:0:(obd_config.c:827:class_cleanup()) Device 5 not setup [ 2243.806413] Lustre: server umount lustre-MDT0000 complete [ 2243.809716] LustreError: 72892:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2252.601519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2252.902093] LustreError: 166-1: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2252.905564] LustreError: 11-0: MGC192.168.202.116@tcp: operation mgs_target_reg to node 0@lo failed: rc = -107 [ 2252.966966] Lustre: Evicted from MGS (at 192.168.202.116@tcp) after server handle changed from 0xf7cec4a4918df9c3 to 0xf7cec4a4918dfa48 [ 2252.996621] Lustre: MGC192.168.202.116@tcp: Connection restored to (at 0@lo) [ 2253.232343] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2253.285882] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2253.721928] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2258.433259] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 2258.453597] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2258.489051] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:353 [ 2258.556126] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 2268.644907] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2268.645400] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2268.659039] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2268.672679] Lustre: Skipped 1 previous similar message [ 2273.761152] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2273.770501] Lustre: Skipped 1 previous similar message [ 2278.881154] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2278.901664] Lustre: Skipped 1 previous similar message [ 2279.393940] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2279.716983] Lustre: server umount lustre-OST0000 complete [ 2284.002895] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2284.014924] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2284.020658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2284.020862] Lustre: Skipped 1 previous similar message [ 2289.121834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2289.130687] Lustre: Skipped 2 previous similar messages [ 2298.335574] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2299.360739] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2299.383625] LustreError: Skipped 15 previous similar messages [ 2300.531822] Lustre: server umount lustre-MDT0000 complete [ 2305.180253] LustreError: 69727:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776194985 with bad export cookie 17856425783666276936 [ 2305.186062] LustreError: 166-1: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2305.206852] LustreError: 69727:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2305.915373] Lustre: server umount lustre-MDT0001 complete [ 2318.871368] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 2321.384741] Key type lgssc unregistered [ 2321.711873] LNet: 74795:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2321.726055] LNet: Removed LNI 192.168.202.116@tcp [ 2322.777650] Key type .llcrypt unregistered [ 2322.779367] Key type ._llcrypt unregistered [ 2342.915498] alg: No test for adler32 (adler32-zlib) [ 2343.671051] Key type ._llcrypt registered [ 2343.674049] Key type .llcrypt registered [ 2343.825830] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 2345.003528] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2345.547753] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 2345.817270] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 2345.821398] LNet: Accept secure, port 988 [ 2347.583274] Key type lgssc registered [ 2348.680539] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2356.542291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2358.096623] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2358.165395] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2362.813220] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 2363.363244] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2368.483580] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2370.685470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2371.090872] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2375.604411] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 2382.420645] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2388.527269] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2395.065734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2395.332703] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2400.236845] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 2407.883333] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2408.627780] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:385 [ 2417.429490] Lustre: Failing over lustre-MDT0000 [ 2417.620122] Lustre: server umount lustre-MDT0000 complete [ 2418.567595] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2419.172293] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2419.191069] Lustre: Skipped 2 previous similar messages [ 2423.668866] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2423.676584] LustreError: Skipped 3 previous similar messages [ 2425.252445] Lustre: 75568:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776195099/real 1776195099] req@00000000e7daf81f x1862475475596672/t0(0) o400->MGC192.168.202.116@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776195106 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2425.271291] LustreError: 166-1: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2425.722814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2431.467378] Lustre: Evicted from MGS (at 192.168.202.116@tcp) after server handle changed from 0x5b920553d2ea97b2 to 0x5b920553d2ea9b9b [ 2431.483779] Lustre: MGC192.168.202.116@tcp: Connection restored to 192.168.202.116@tcp (at 0@lo) [ 2431.622723] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2431.707401] LustreError: 79414:0:(obd_config.c:774:class_setup()) setup lustre-MDT0000 failed (-9) [ 2431.713805] LustreError: 79414:0:(obd_config.c:1999:class_config_llog_handler()) MGC192.168.202.116@tcp: cfg command failed: rc = -9 [ 2431.727850] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2431.727850] [ 2431.737235] LustreError: 15c-8: MGC192.168.202.116@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2431.752478] LustreError: 79401:0:(obd_mount_server.c:1425:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2431.766612] LustreError: 79401:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -9 [ 2431.782566] LustreError: 79401:0:(obd_config.c:827:class_cleanup()) Device 5 not setup [ 2431.825988] Lustre: server umount lustre-MDT0000 complete [ 2431.835545] LustreError: 79401:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2431.856876] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2431.863901] LustreError: Skipped 14 previous similar messages [ 2438.304714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2438.409816] LustreError: 11-0: MGC192.168.202.116@tcp: operation mgs_target_reg to node 0@lo failed: rc = -107 [ 2438.416099] LustreError: 166-1: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2438.430371] Lustre: Evicted from MGS (at 192.168.202.116@tcp) after server handle changed from 0x5b920553d2ea9b9b to 0x5b920553d2ea9c2e [ 2438.440190] Lustre: MGC192.168.202.116@tcp: Connection restored to 192.168.202.116@tcp (at 0@lo) [ 2438.654488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2438.698720] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2439.977482] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2442.842209] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 2443.756974] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 2443.780675] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2443.818502] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:417 [ 2452.960136] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2452.968155] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2452.992388] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2453.991568] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2453.993159] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2457.525671] Lustre: server umount lustre-OST0000 complete [ 2464.228115] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2464.235293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2464.247641] Lustre: Skipped 1 previous similar message [ 2464.269338] Lustre: Skipped 2 previous similar messages [ 2468.532943] Lustre: server umount lustre-MDT0000 complete [ 2469.344900] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2469.369729] LustreError: Skipped 9 previous similar messages [ 2473.852070] LustreError: 76240:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776195154 with bad export cookie 6598342261630213166 [ 2473.861045] LustreError: 166-1: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2473.872231] LustreError: 76240:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2474.313553] Lustre: server umount lustre-MDT0001 complete [ 2485.678848] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 2488.385077] Key type lgssc unregistered [ 2488.692839] LNet: 81300:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2488.714466] LNet: Removed LNI 192.168.202.116@tcp [ 2489.444969] Key type .llcrypt unregistered [ 2489.447176] Key type ._llcrypt unregistered [ 2514.014967] alg: No test for adler32 (adler32-zlib) [ 2514.772274] Key type ._llcrypt registered [ 2514.777699] Key type .llcrypt registered [ 2514.864920] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 2516.095870] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2516.805520] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 2517.120845] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 2517.137223] LNet: Accept secure, port 988 [ 2518.855165] Key type lgssc registered [ 2520.108885] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2528.254969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2529.735071] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2529.825647] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2534.022566] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 2534.882208] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2540.002564] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2543.245204] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2543.563043] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2550.158065] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 2561.286501] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2567.246690] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2575.232560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2575.866527] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2579.973785] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:449 [ 2581.381217] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 2588.982486] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2603.495523] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2603.498249] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2603.539218] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2603.556080] Lustre: Skipped 1 previous similar message [ 2605.198703] Lustre: server umount lustre-OST0000 complete [ 2610.145539] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2610.162058] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2610.175344] Lustre: Skipped 1 previous similar message [ 2610.180218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2613.736833] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2613.751126] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2616.214625] Lustre: server umount lustre-MDT0000 complete [ 2618.850029] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2618.874863] LustreError: Skipped 1 previous similar message [ 2621.403861] LustreError: 82768:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776195302 with bad export cookie 2946147631705381984 [ 2621.417833] LustreError: 166-1: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2621.429674] LustreError: 82768:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2621.910765] Lustre: server umount lustre-MDT0001 complete [ 2633.550926] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 2636.710465] Key type lgssc unregistered [ 2636.986218] LNet: 86461:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2636.995912] LNet: Removed LNI 192.168.202.116@tcp [ 2637.863407] Key type .llcrypt unregistered [ 2637.867526] Key type ._llcrypt unregistered [ 2659.985955] alg: No test for adler32 (adler32-zlib) [ 2660.743105] Key type ._llcrypt registered [ 2660.752583] Key type .llcrypt registered [ 2660.913602] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 2662.083511] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2662.939480] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 2663.266125] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 2663.270256] LNet: Accept secure, port 988 [ 2665.000120] Key type lgssc registered [ 2666.272442] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2675.447605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2677.212688] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2677.317587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2682.344149] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2682.591639] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 2687.459986] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2692.179993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2692.595747] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2692.855833] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2697.788432] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 2706.278546] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2713.947816] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2721.736994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2722.100505] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2727.168232] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 2728.280871] Lustre: lustre-OST0000: deleting orphan objects from 0x0:451 to 0x0:481 [ 2736.144262] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2743.781359] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2743.784073] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2743.786614] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2743.797249] LustreError: Skipped 1 previous similar message [ 2743.832876] Lustre: Skipped 1 previous similar message [ 2744.451398] Lustre: server umount lustre-OST0000 complete [ 2748.896662] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2748.912688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2748.918062] Lustre: Skipped 1 previous similar message [ 2748.919560] Lustre: Skipped 1 previous similar message [ 2754.019614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2754.023019] Lustre: Skipped 2 previous similar messages [ 2759.136078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2763.231297] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2763.360902] Lustre: server umount lustre-MDT0000 complete [ 2765.282325] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2765.296356] LustreError: Skipped 1 previous similar message [ 2767.222676] LustreError: 87905:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776195447 with bad export cookie 15988701773536907534 [ 2767.233317] LustreError: 166-1: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2767.245586] LustreError: 87905:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2767.526063] Lustre: server umount lustre-MDT0001 complete [ 2777.769863] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 2779.988664] Key type lgssc unregistered [ 2780.234472] LNet: 91582:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2780.243427] LNet: Removed LNI 192.168.202.116@tcp [ 2780.813233] Key type .llcrypt unregistered [ 2780.817350] Key type ._llcrypt unregistered [ 2801.687195] alg: No test for adler32 (adler32-zlib) [ 2802.439827] Key type ._llcrypt registered [ 2802.442607] Key type .llcrypt registered [ 2802.548394] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 2803.302802] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2804.053768] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 2804.235832] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 2804.239411] LNet: Accept secure, port 988 [ 2805.887345] Key type lgssc registered [ 2806.942596] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2815.855081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2817.231950] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2817.304059] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2822.300655] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 2822.638864] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2827.747681] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2831.437256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2831.890592] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2837.049682] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 2844.754612] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2851.531297] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2858.292336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2858.772713] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2863.318611] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 2869.071653] Lustre: lustre-OST0000: deleting orphan objects from 0x0:451 to 0x0:513 [ 2870.600347] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2882.553947] Lustre: server umount lustre-OST0000 complete [ 2888.160913] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2888.175853] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2888.202857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2889.696177] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2889.697616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2889.697766] Lustre: 92359:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776195563/real 1776195563] req@00000000a278a6c9 x1862475956893184/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1776195570 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 2889.706809] Lustre: Skipped 1 previous similar message [ 2889.740895] Lustre: Skipped 1 previous similar message [ 2891.744840] Lustre: 92358:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776195565/real 1776195565] req@00000000ecc19704 x1862475956893632/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1776195572 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2892.767237] Lustre: 95898:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776195567/real 1776195567] req@00000000295cf689 x1862475956894336/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1776195573 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2893.013457] Lustre: server umount lustre-MDT0000 complete [ 2894.821181] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2894.839225] LustreError: Skipped 1 previous similar message [ 2896.511871] LustreError: 93026:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776195577 with bad export cookie 1424835577732755621 [ 2896.514243] LustreError: 166-1: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2896.520022] LustreError: 93026:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2896.885398] Lustre: server umount lustre-MDT0001 complete [ 2910.603338] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 2914.346923] Key type lgssc unregistered [ 2914.686469] LNet: 96703:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2914.723434] LNet: Removed LNI 192.168.202.116@tcp [ 2915.805856] Key type .llcrypt unregistered [ 2915.808524] Key type ._llcrypt unregistered [ 2940.868993] alg: No test for adler32 (adler32-zlib) [ 2941.632765] Key type ._llcrypt registered [ 2941.639995] Key type .llcrypt registered [ 2941.791909] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 2943.168793] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2943.954586] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 2944.209704] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 2944.212994] LNet: Accept secure, port 988 [ 2945.863689] Key type lgssc registered [ 2947.324364] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2954.859988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2972.575464] LustreError: 98158:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.116@tcp: failed processing log, type 1: rc = -5 [ 3003.295959] LustreError: 98158:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.116@tcp: failed processing log, type 4: rc = -110 [ 3033.055280] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3037.587709] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 3045.430748] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3053.063180] Lustre: server umount lustre-OST0000 complete [ 3061.099451] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 15:42:20 (1776195740) [ 3068.547241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3068.896328] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3068.951256] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3074.024095] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3074.267552] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 3079.154236] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3083.223911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3083.233307] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3083.697821] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3088.918684] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 3096.493772] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3102.856277] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3110.538385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3110.930144] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3116.596946] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 3119.162647] Lustre: lustre-OST0000: deleting orphan objects from 0x0:516 to 0x0:545 [ 3124.846470] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3139.553549] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3139.557211] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3139.577484] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3139.592933] Lustre: Skipped 1 previous similar message [ 3139.614797] Lustre: Skipped 1 previous similar message [ 3141.081072] Lustre: server umount lustre-OST0000 complete [ 3149.792821] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3149.805285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3149.810591] Lustre: Skipped 1 previous similar message [ 3151.987432] Lustre: server umount lustre-MDT0000 complete [ 3154.918791] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3154.938478] LustreError: Skipped 1 previous similar message [ 3156.707759] LustreError: 99740:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776195837 with bad export cookie 850482486267549505 [ 3156.732456] LustreError: 166-1: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3157.257741] Lustre: server umount lustre-MDT0001 complete [ 3171.044912] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 3174.064911] Key type lgssc unregistered [ 3174.433663] LNet: 103544:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3174.439121] LNet: Removed LNI 192.168.202.116@tcp [ 3175.188569] Key type .llcrypt unregistered [ 3175.190640] Key type ._llcrypt unregistered [ 3196.526714] alg: No test for adler32 (adler32-zlib) [ 3197.284712] Key type ._llcrypt registered [ 3197.290848] Key type .llcrypt registered [ 3197.421987] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 3198.354592] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3198.939532] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 3199.092728] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 3199.101764] LNet: Accept secure, port 988 [ 3200.767177] Key type lgssc registered [ 3201.916909] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3209.544429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3210.865120] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3210.901853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3215.465488] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 3216.354510] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3221.479182] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3224.327504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3224.585841] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3229.240389] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 3236.640260] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3243.280191] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3249.353694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3249.622597] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3254.846163] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 3261.940247] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3261.955352] Lustre: lustre-OST0000: deleting orphan objects from 0x0:516 to 0x0:577 [ 3270.113688] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3270.120419] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3270.142249] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3272.177321] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3272.177722] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3272.178597] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3272.187201] Lustre: Skipped 1 previous similar message [ 3275.823562] Lustre: server umount lustre-OST0000 complete [ 3280.868124] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3280.875897] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3280.891961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3285.792187] Lustre: server umount lustre-MDT0000 complete [ 3287.527661] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3287.554469] LustreError: Skipped 1 previous similar message [ 3289.459727] LustreError: 104987:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776195970 with bad export cookie 3707443594829877442 [ 3289.473494] LustreError: 166-1: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3289.790581] Lustre: server umount lustre-MDT0001 complete [ 3301.812290] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 3304.275049] Key type lgssc unregistered [ 3304.545026] LNet: 108669:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3304.553183] LNet: Removed LNI 192.168.202.116@tcp [ 3305.217670] Key type .llcrypt unregistered [ 3305.219660] Key type ._llcrypt unregistered [ 3319.868630] alg: No test for adler32 (adler32-zlib) [ 3320.639066] Key type ._llcrypt registered [ 3320.646539] Key type .llcrypt registered [ 3320.769757] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 3321.639479] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3322.167446] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 3322.342975] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 3322.345978] LNet: Accept secure, port 988 [ 3324.015066] Key type lgssc registered [ 3324.808241] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3331.163477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3348.767826] LustreError: 109962:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.116@tcp: failed processing log, type 1: rc = -5 [ 3379.487315] LustreError: 109962:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.116@tcp: failed processing log, type 4: rc = -110 [ 3410.335211] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3415.732447] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 3422.501258] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3428.275544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3432.082538] LustreError: 15c-8: MGC192.168.202.116@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3432.107703] LustreError: 110948:0:(obd_mount_server.c:1425:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3432.120210] LustreError: 110948:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -2 [ 3432.136857] LustreError: 110948:0:(obd_mount_server.c:1644:server_put_super()) no obd lustre-MDT0000 [ 3432.152855] LustreError: 109969:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776196112 with bad export cookie 6102630878373980 [ 3432.172576] LustreError: 166-1: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3432.339587] Lustre: server umount lustre-MDT0000 complete [ 3432.345783] LustreError: 110948:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3458.221121] Lustre: server umount lustre-OST0000 complete [ 3468.986426] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 3478.654407] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 3489.326783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3495.638609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3501.217107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3507.858543] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3514.789543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3514.899865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3515.192975] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3515.238485] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3515.367376] Lustre: lustre-MDT0000: new disk, initializing [ 3515.450680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3515.480609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3521.181738] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 3531.943787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3532.096350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3532.242847] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3532.300813] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3532.309843] Lustre: Skipped 1 previous similar message [ 3532.427372] Lustre: lustre-MDT0001: new disk, initializing [ 3532.505930] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3532.530479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3532.542632] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3537.622435] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 3546.699338] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3553.021762] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3560.956984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3561.075529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3561.092222] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3561.516719] Lustre: lustre-OST0000: new disk, initializing [ 3561.521511] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3561.660107] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3567.208893] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 3571.804240] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3571.821416] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3577.260930] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3582.349642] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3582.667644] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3587.194627] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3587.402141] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3592.673762] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3592.690974] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3592.710810] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3595.503950] Lustre: server umount lustre-OST0000 complete [ 3599.328415] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3599.334194] LustreError: Skipped 1 previous similar message [ 3599.346295] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3599.354025] Lustre: Skipped 1 previous similar message [ 3599.356880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3599.360207] Lustre: Skipped 1 previous similar message [ 3602.913542] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3602.920233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3602.936555] Lustre: Skipped 1 previous similar message [ 3605.496323] Lustre: server umount lustre-MDT0000 complete [ 3608.032541] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3608.055154] LustreError: Skipped 1 previous similar message [ 3609.058840] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3609.072856] LustreError: Skipped 1 previous similar message [ 3609.315705] LustreError: 113830:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776196290 with bad export cookie 6102630878374078 [ 3609.316783] LustreError: 166-1: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3609.322545] LustreError: 113830:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3609.683949] Lustre: server umount lustre-MDT0001 complete [ 3618.398899] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 15:51:37 (1776196297) [ 3622.071259] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3640.816175] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 3650.246744] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 3660.546906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3667.641281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3673.391090] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3678.932939] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3686.234808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3686.296650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3686.668836] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3686.693782] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3686.785735] Lustre: lustre-MDT0000: new disk, initializing [ 3686.941517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3686.971609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3692.023035] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 3704.221989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3704.345622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3704.491893] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3704.537356] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3704.542539] Lustre: Skipped 1 previous similar message [ 3704.645874] Lustre: lustre-MDT0001: new disk, initializing [ 3704.815545] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3704.880511] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3704.899248] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3710.427373] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 3722.619132] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3730.378924] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3738.753291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3738.847128] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3738.855692] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3739.147659] Lustre: lustre-OST0000: new disk, initializing [ 3739.151874] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3739.231517] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3744.441493] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 3749.445384] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3749.464346] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3754.239718] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3758.772910] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3759.010992] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3764.632546] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3764.954191] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3770.341534] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3770.342619] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3770.348887] LustreError: Skipped 1 previous similar message [ 3770.353770] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3770.391546] Lustre: Skipped 1 previous similar message [ 3773.451355] Lustre: server umount lustre-OST0000 complete [ 3780.582884] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3780.590384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3780.598280] Lustre: Skipped 1 previous similar message [ 3780.611151] Lustre: Skipped 1 previous similar message [ 3791.328297] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3791.458309] Lustre: server umount lustre-MDT0000 complete [ 3791.841461] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3791.863420] LustreError: Skipped 1 previous similar message [ 3794.563427] LustreError: 120966:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776196475 with bad export cookie 6102630878374932 [ 3794.570346] LustreError: 120966:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3794.571895] LustreError: 166-1: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3802.576778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3802.962536] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3803.044820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3806.813639] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 3808.236608] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3808.259936] LustreError: Skipped 1 previous similar message [ 3814.847838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3819.704439] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 3826.695547] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3833.167322] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3839.739885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3846.019496] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 3853.999703] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3861.489367] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3868.336033] Lustre: server umount lustre-OST0000 complete [ 3868.343864] Lustre: Skipped 1 previous similar message [ 3873.760754] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3873.769850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3873.793313] Lustre: Skipped 1 previous similar message [ 3873.818176] Lustre: Skipped 6 previous similar messages [ 3875.807543] Lustre: 109302:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776196549/real 1776196549] req@000000009164ef3f x1862476500099328/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1776196556 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3877.471144] Lustre: 109300:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776196551/real 1776196551] req@0000000089098243 x1862476500099456/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1776196558 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 3879.903869] Lustre: 128274:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776196554/real 1776196554] req@000000005f645015 x1862476500100096/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1776196560 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3880.932219] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3880.949894] LustreError: Skipped 2 previous similar messages [ 3883.921226] LustreError: 125313:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776196564 with bad export cookie 6102630878375779 [ 3883.922304] LustreError: 166-1: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3883.931802] LustreError: 125313:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 3896.186332] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 3898.788871] Key type lgssc unregistered [ 3899.183499] LNet: 129080:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3899.188892] LNet: Removed LNI 192.168.202.116@tcp [ 3900.110880] Key type .llcrypt unregistered [ 3900.115814] Key type ._llcrypt unregistered [ 3917.969695] alg: No test for adler32 (adler32-zlib) [ 3918.726332] Key type ._llcrypt registered [ 3918.735206] Key type .llcrypt registered [ 3918.886153] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 3931.624677] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 3932.829827] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3933.308491] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 3933.519922] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 3933.524937] LNet: Accept secure, port 988 [ 3935.263243] Key type lgssc registered [ 3936.392901] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3946.875569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3954.119818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3961.501528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3967.562255] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3974.264547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3974.322438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3975.647990] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3975.713621] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3975.827128] Lustre: lustre-MDT0000: new disk, initializing [ 3975.919598] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3975.939568] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3979.968829] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 3989.191857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3989.261351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3989.327706] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3989.361478] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3989.368508] Lustre: Skipped 1 previous similar message [ 3989.441129] Lustre: lustre-MDT0001: new disk, initializing [ 3989.508435] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3989.551179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3989.570108] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3993.424870] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 4002.251503] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4007.847709] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4015.221917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4015.342922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4015.363082] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4015.739866] Lustre: lustre-OST0000: new disk, initializing [ 4015.745212] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 4015.850165] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4022.636396] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 4023.432684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4023.443116] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4032.673178] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4037.691988] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 4037.903771] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4044.283089] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 4044.701885] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4049.380511] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4049.416286] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4049.433982] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4049.433982] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4053.787689] Lustre: server umount lustre-OST0000 complete [ 4059.090454] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4059.098039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4059.126305] Lustre: Skipped 2 previous similar messages [ 4059.158452] Lustre: Skipped 1 previous similar message [ 4064.224910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4064.250312] Lustre: Skipped 1 previous similar message [ 4065.458708] Lustre: server umount lustre-MDT0000 complete [ 4070.369060] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4070.384276] LustreError: Skipped 1 previous similar message [ 4070.943634] LustreError: 132354:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776196751 with bad export cookie 3427292863320551902 [ 4070.946159] LustreError: 166-1: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4070.957572] LustreError: 132354:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4071.384477] Lustre: server umount lustre-MDT0001 complete [ 4081.829670] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 15:59:20 (1776196760) [ 4088.786624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4089.150129] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4089.229858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4094.443786] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4095.142476] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 4099.555047] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4104.529233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4104.674313] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4105.005980] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4110.219722] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 4120.585681] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4127.037864] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4129.264838] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4129.280640] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4129.284625] Lustre: Skipped 1 previous similar message [ 4129.316655] Lustre: Skipped 3 previous similar messages [ 4134.371750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4134.378745] Lustre: Skipped 1 previous similar message [ 4135.574078] Lustre: server umount lustre-MDT0000 complete [ 4139.018372] LustreError: 138593:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776196819 with bad export cookie 3427292863320552756 [ 4139.026618] LustreError: 166-1: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4139.029113] LustreError: 138593:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4147.020417] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 16:00:26 (1776196826) [ 4153.193551] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4170.790522] LustreError: 139438:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.116@tcp: failed processing log, type 1: rc = -5 [ 4181.023508] LustreError: 139438:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.116@tcp: failed processing log, type 4: rc = -110 [ 4210.847439] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4215.640405] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 4224.313775] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4226.498053] Lustre: server umount lustre-OST0000 complete [ 4226.501767] Lustre: Skipped 1 previous similar message [ 4234.685795] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 16:01:53 (1776196913) [ 4241.729907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4242.117577] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4242.190148] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4247.562088] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 4256.062312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4261.639174] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 4269.225908] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4275.650587] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4282.768298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4283.133391] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4283.146476] Lustre: Skipped 1 previous similar message [ 4287.850987] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 4296.027589] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4307.424830] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4307.436095] LustreError: Skipped 1 previous similar message [ 4307.438170] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4307.454795] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4307.457331] Lustre: Skipped 2 previous similar messages [ 4309.175824] Lustre: server umount lustre-MDT0000 complete [ 4313.066966] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4313.090858] LustreError: Skipped 4 previous similar messages [ 4313.219891] LustreError: 140837:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776196993 with bad export cookie 3427292863320553421 [ 4313.233887] LustreError: 166-1: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4313.238090] LustreError: 140837:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4323.295748] Lustre: 143907:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776196998/real 1776196998] req@00000000ad581afa x1862477140767232/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776197004 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4332.054364] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 16:03:31 (1776197011) [ 4339.210535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4345.483712] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 4349.926990] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4349.969681] LustreError: Skipped 2 previous similar messages [ 4354.997592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4355.229406] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4355.234158] Lustre: Skipped 1 previous similar message [ 4359.803841] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 4367.784284] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4373.954909] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4380.328746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4385.800180] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 4389.807819] Lustre: lustre-OST0000: deleting orphan objects from 0x0:36 to 0x0:65 [ 4393.750601] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4398.070449] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 4398.300650] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4402.404259] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 4402.567969] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4405.223205] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4405.238329] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4405.243799] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4405.257589] Lustre: Skipped 3 previous similar messages [ 4405.272342] Lustre: Skipped 3 previous similar messages [ 4410.683577] Lustre: server umount lustre-OST0000 complete [ 4410.686906] Lustre: Skipped 2 previous similar messages [ 4415.469822] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4415.493589] Lustre: Skipped 1 previous similar message [ 4425.266083] LustreError: 144559:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776197106 with bad export cookie 3427292863320554492 [ 4425.277262] LustreError: 166-1: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4425.281099] LustreError: 144559:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4434.413460] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 16:05:13 (1776197113) [ 4442.282828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4459.039561] LustreError: 148907:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.116@tcp: failed processing log, type 1: rc = -5 [ 4470.303444] LustreError: 148907:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.116@tcp: failed processing log, type 4: rc = -110 [ 4499.103301] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4499.120962] Lustre: Skipped 1 previous similar message [ 4503.267908] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 4510.011598] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4517.074380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4517.681673] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4517.697736] LustreError: Skipped 1 previous similar message [ 4517.815323] Lustre: lustre-OST0000: deleting orphan objects from 0x0:36 to 0x0:97 [ 4522.009198] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 4530.735369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4535.547240] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 4542.693744] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4548.983994] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4553.461445] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 4553.741142] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4557.834618] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 4558.033402] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4561.895208] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4561.898980] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4561.903164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4561.903175] Lustre: Skipped 6 previous similar messages [ 4561.917882] Lustre: Skipped 1 previous similar message [ 4568.381942] Lustre: server umount lustre-MDT0000 complete [ 4568.395904] Lustre: Skipped 2 previous similar messages [ 4572.877280] LustreError: 148915:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776197253 with bad export cookie 3427292863320555514 [ 4572.881518] LustreError: 166-1: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4572.900606] LustreError: 148915:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4582.367184] Lustre: 152621:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776197257/real 1776197257] req@0000000086ecb091 x1862477140797184/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776197263 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4591.009586] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 16:07:50 (1776197270) [ 4598.089288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4615.775532] LustreError: 153265:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.116@tcp: failed processing log, type 1: rc = -5 [ 4626.015595] LustreError: 153265:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.116@tcp: failed processing log, type 4: rc = -110 [ 4660.716503] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 4668.272679] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4675.121687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4695.655576] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4695.675915] LustreError: Skipped 12 previous similar messages [ 4699.103903] Lustre: lustre-OST0000: deleting orphan objects from 0x0:36 to 0x0:129 [ 4703.377088] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 4710.907783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4714.748702] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 4720.686924] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4726.317180] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4734.007994] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4734.139059] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4734.149852] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 4734.381968] Lustre: lustre-OST0001: new disk, initializing [ 4734.385539] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 4739.676639] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 4740.672249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4740.683378] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4751.820814] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4756.430910] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 4756.706349] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4760.916260] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 4761.079113] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4765.671092] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4765.677510] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4765.699301] Lustre: Skipped 1 previous similar message [ 4765.709546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4765.712323] Lustre: Skipped 2 previous similar messages [ 4769.667030] Lustre: server umount lustre-OST0000 complete [ 4769.678866] Lustre: Skipped 2 previous similar messages [ 4776.939847] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4797.149418] LustreError: 153271:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776197477 with bad export cookie 3427292863320556466 [ 4797.159053] LustreError: 166-1: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4797.183394] LustreError: 153271:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4805.177502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4808.815176] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4813.070549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4816.694683] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4823.026166] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 16:11:42 (1776197502) [ 4824.628474] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4826.520346] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4828.831394] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 16:11:47 (1776197507) [ 4834.745672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4835.007783] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4835.054633] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4835.284469] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4835.291702] Lustre: Skipped 6 previous similar messages [ 4840.586687] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 4850.811967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4851.130648] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4851.205056] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4857.798089] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 4867.853229] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4874.012574] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4879.527147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4879.685745] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4883.374900] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 4887.691248] Lustre: lustre-OST0000: deleting orphan objects from 0x0:36 to 0x0:161 [ 4890.670839] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4894.868513] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 4895.176111] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4900.371802] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 4900.666788] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4902.883447] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4902.887455] LustreError: Skipped 1 previous similar message [ 4902.889708] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4902.901498] Lustre: Skipped 5 previous similar messages [ 4902.911069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4902.915684] Lustre: Skipped 8 previous similar messages [ 4908.611575] Lustre: server umount lustre-OST0000 complete [ 4908.614551] Lustre: Skipped 3 previous similar messages [ 4933.036594] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4935.406230] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4935.417047] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to (at 0@lo) [ 4935.428986] Lustre: lustre-OST0000: deleting orphan objects from 0x0:36 to 0x0:193 [ 4937.723391] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 4945.037271] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4945.633214] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4945.660598] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.202.116@tcp (at 0@lo) [ 4945.668594] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:162 to 0x280000400:193 [ 4950.484815] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 4950.707900] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4954.986774] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 4955.266080] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4960.299439] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff905e12589800.ost_server_uuid 40 [ 4961.255464] Lustre: lustre-MDT0001: haven't heard from client 1778c3b1-b252-460f-9275-203c8be9b161 (at 192.168.202.16@tcp) in 50 seconds. I think it's dead, and I am evicting it. exp 000000001af74c38, cur 1776197642 expire 1776197612 last 1776197592 [ 4962.096691] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff905e12589800.ost_server_uuid in FULL state after 0 sec [ 4966.376646] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4966.399433] LustreError: Skipped 2 previous similar messages [ 4985.533203] LustreError: 159838:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776197666 with bad export cookie 3427292863320557621 [ 4985.534677] LustreError: 166-1: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4985.552257] LustreError: 159838:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4996.492880] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 4998.897688] Key type lgssc unregistered [ 4999.164870] LNet: 166239:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4999.180196] LNet: Removed LNI 192.168.202.116@tcp [ 4999.917102] Key type .llcrypt unregistered [ 4999.918729] Key type ._llcrypt unregistered [ 5022.901077] alg: No test for adler32 (adler32-zlib) [ 5023.657884] Key type ._llcrypt registered [ 5023.660266] Key type .llcrypt registered [ 5023.778554] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 5024.561074] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5024.965899] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 5025.143357] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 5025.146920] LNet: Accept secure, port 988 [ 5026.783101] Key type lgssc registered [ 5027.807772] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5034.917510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5036.448882] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5036.540228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5041.560535] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 5041.634115] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5046.754555] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5050.557763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5050.875649] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5055.434923] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 5062.338525] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5068.352626] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5074.696499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5074.961960] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5079.173238] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 5085.419780] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5086.263318] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:225 [ 5095.326788] Lustre: Failing over lustre-MDT0000 [ 5095.541080] Lustre: server umount lustre-MDT0000 complete [ 5096.339142] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5096.421804] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5096.440129] Lustre: Skipped 2 previous similar messages [ 5101.536397] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5101.556101] LustreError: Skipped 5 previous similar messages [ 5103.455147] Lustre: 167016:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776197777/real 1776197777] req@000000000da53a97 x1862478284731392/t0(0) o400->MGC192.168.202.116@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776197784 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 5103.476518] LustreError: 166-1: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5109.732818] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5109.760609] LustreError: Skipped 8 previous similar messages [ 5114.706831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5120.995842] Lustre: Evicted from MGS (at 192.168.202.116@tcp) after server handle changed from 0xc76c7c50b759ec20 to 0xc76c7c50b759f002 [ 5121.020480] Lustre: MGC192.168.202.116@tcp: Connection restored to 192.168.202.116@tcp (at 0@lo) [ 5121.290438] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5121.355690] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5126.143822] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 5126.629589] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5126.640508] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 5128.405927] Lustre: lustre-MDT0000: Denying connection for new client 6a355109-395c-4417-8d3b-43cc3ee794c9 (at 192.168.202.16@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:03 [ 5136.881522] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5136.907660] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5136.919226] Lustre: Skipped 1 previous similar message [ 5141.992890] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5141.998037] Lustre: Skipped 1 previous similar message [ 5142.173734] Lustre: server umount lustre-OST0000 complete [ 5145.639912] LustreError: 171559:0:(ldlm_lib.c:2887:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5145.647936] Lustre: 170855:0:(ldlm_lib.c:2290:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5145.653746] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5145.674054] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5145.685822] LustreError: 170855:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000029570382 x1862478284741696/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' [ 5145.698492] LustreError: 170855:0:(fid_request.c:233:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5145.704754] LustreError: 170855:0:(fid_request.c:335:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5145.712037] Lustre: lustre-MDT0000: Recovery over after 0:19, of 2 clients 0 recovered and 2 were evicted. [ 5145.740335] Lustre: 170855:0:(mdt_handler.c:7538:mdt_postrecov()) lustre-MDT0000: auto trigger paused LFSCK failed: rc = -6 [ 5145.943121] Lustre: server umount lustre-MDT0000 complete [ 5147.107171] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5147.121216] Lustre: Skipped 1 previous similar message [ 5147.132235] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5147.155197] LustreError: Skipped 12 previous similar messages [ 5149.610278] LustreError: 167684:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776197830 with bad export cookie 14369997197147762690 [ 5149.618914] LustreError: 166-1: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5149.629890] LustreError: 167012:0:(import.c:692:ptlrpc_connect_import_locked()) can't connect to a closed import [ 5152.018957] Lustre: server umount lustre-MDT0001 complete [ 5167.791206] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 5171.073925] Key type lgssc unregistered [ 5171.306665] LNet: 172364:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5171.329246] LNet: Removed LNI 192.168.202.116@tcp [ 5172.153902] Key type .llcrypt unregistered [ 5172.156622] Key type ._llcrypt unregistered [ 5195.269933] alg: No test for adler32 (adler32-zlib) [ 5196.020682] Key type ._llcrypt registered [ 5196.023851] Key type .llcrypt registered [ 5196.149995] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 5197.590660] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5198.090962] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 5198.333157] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 5198.341649] LNet: Accept secure, port 988 [ 5199.999129] Key type lgssc registered [ 5201.273258] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5209.341298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5210.823847] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5210.922885] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5215.264643] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 5216.225455] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5221.345941] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5223.849370] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5224.179736] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5228.501839] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 5236.519544] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5241.989629] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5249.664600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5250.022171] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5255.364614] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 5262.344494] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:257 [ 5263.070470] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5272.547038] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5272.555433] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5272.573385] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5272.582529] Lustre: Skipped 1 previous similar message [ 5272.598587] Lustre: Skipped 1 previous similar message [ 5275.370102] Lustre: server umount lustre-OST0000 complete [ 5280.742977] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5280.762486] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5280.784529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5282.785458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5282.788605] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5285.713600] Lustre: server umount lustre-MDT0000 complete [ 5287.907597] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5287.935322] LustreError: Skipped 1 previous similar message [ 5289.726038] LustreError: 173811:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776197970 with bad export cookie 8846143294118180565 [ 5289.730758] LustreError: 166-1: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5289.731650] LustreError: 173811:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5290.145935] Lustre: server umount lustre-MDT0001 complete [ 5303.277117] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 5306.874658] Key type lgssc unregistered [ 5307.266748] LNet: 177484:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5307.276611] LNet: Removed LNI 192.168.202.116@tcp [ 5308.534775] Key type .llcrypt unregistered [ 5308.539464] Key type ._llcrypt unregistered [ 5342.400672] alg: No test for adler32 (adler32-zlib) [ 5343.155545] Key type ._llcrypt registered [ 5343.157365] Key type .llcrypt registered [ 5343.264292] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 5344.078022] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5344.497564] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 5344.653607] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 5344.656573] LNet: Accept secure, port 988 [ 5346.319290] Key type lgssc registered [ 5347.194697] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5354.474155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5356.005400] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5356.071540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5360.408688] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 5361.124383] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5366.240530] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5368.373655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5368.588370] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5372.838850] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 5380.335566] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5385.552762] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5392.503401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5392.877208] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5399.424760] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 5406.188144] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:289 [ 5410.609707] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5419.458635] Lustre: server umount lustre-OST0000 complete [ 5425.124088] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5425.136070] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5425.149146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5426.657579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5426.658908] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5427.743291] Lustre: 178551:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776198100/real 1776198100] req@0000000022a5b7ca x1862478620276800/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1776198107 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 5427.767935] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5427.781078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5427.786101] Lustre: Skipped 2 previous similar messages [ 5428.831509] Lustre: 178552:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776198102/real 1776198102] req@0000000096b7788a x1862478620277248/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1776198109 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 5429.394828] Lustre: server umount lustre-MDT0000 complete [ 5432.806421] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5432.815071] LustreError: Skipped 1 previous similar message [ 5433.276126] LustreError: 179221:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776198114 with bad export cookie 2352185165352661087 [ 5433.284035] LustreError: 166-1: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5433.291033] LustreError: 179221:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5433.621800] Lustre: server umount lustre-MDT0001 complete [ 5445.097285] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 5447.399956] Key type lgssc unregistered [ 5447.679475] LNet: 182894:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5447.686790] LNet: Removed LNI 192.168.202.116@tcp [ 5448.465824] Key type .llcrypt unregistered [ 5448.468746] Key type ._llcrypt unregistered [ 5465.521922] alg: No test for adler32 (adler32-zlib) [ 5466.270321] Key type ._llcrypt registered [ 5466.271401] Key type .llcrypt registered [ 5466.391883] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 5467.213401] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5467.803851] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 5467.951131] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 5467.954809] LNet: Accept secure, port 988 [ 5469.615429] Key type lgssc registered [ 5470.695589] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5480.668467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5482.076722] Lustre: *** cfs_fail_loc=135, val=0*** [ 5482.176882] LustreError: 184466:0:(obd_config.c:774:class_setup()) setup lustre-MDT0000 failed (-2) [ 5482.188896] LustreError: 184466:0:(obd_config.c:1999:class_config_llog_handler()) MGC192.168.202.116@tcp: cfg command failed: rc = -2 [ 5482.195880] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5482.195880] [ 5482.220994] LustreError: 15c-8: MGC192.168.202.116@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5482.250274] LustreError: 184427:0:(obd_mount_server.c:1425:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5482.265313] LustreError: 184427:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -2 [ 5482.277776] LustreError: 184427:0:(obd_config.c:827:class_cleanup()) Device 5 not setup [ 5488.610340] Lustre: 184427:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776198163/real 1776198163] req@00000000e10bfcd4 x1862478749238080/t0(0) o251->MGC192.168.202.116@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776198169 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 5488.747678] Lustre: server umount lustre-MDT0000 complete [ 5488.749688] LustreError: 184427:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5496.128353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5496.623866] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5496.682946] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5500.265363] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 5501.934836] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5507.042823] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5507.117732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5507.430350] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5511.823835] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 5518.922347] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5526.473744] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5532.128809] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5532.137119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5532.148347] Lustre: Skipped 1 previous similar message [ 5535.475878] Lustre: server umount lustre-MDT0000 complete [ 5537.268208] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5537.286500] LustreError: Skipped 1 previous similar message [ 5539.172171] LustreError: 184820:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776198219 with bad export cookie 10519704960283970217 [ 5539.175538] LustreError: 166-1: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5539.191627] LustreError: 184820:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5539.532975] Lustre: server umount lustre-MDT0001 complete [ 5550.256556] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 5553.217129] Key type lgssc unregistered [ 5553.600674] LNet: 187320:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5553.614534] LNet: Removed LNI 192.168.202.116@tcp [ 5554.469465] Key type .llcrypt unregistered [ 5554.474575] Key type ._llcrypt unregistered [ 5575.417360] alg: No test for adler32 (adler32-zlib) [ 5576.174379] Key type ._llcrypt registered [ 5576.176154] Key type .llcrypt registered [ 5576.270608] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 5576.452329] Key type .llcrypt unregistered [ 5576.454351] Key type ._llcrypt unregistered [ 5590.324610] alg: No test for adler32 (adler32-zlib) [ 5591.077790] Key type ._llcrypt registered [ 5591.079191] Key type .llcrypt registered [ 5591.185150] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 5592.558348] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5593.038496] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 5593.239113] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 5593.248868] LNet: Accept secure, port 988 [ 5594.959105] Key type lgssc registered [ 5596.060867] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5602.387100] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5619.935395] LustreError: 189576:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.116@tcp: failed processing log, type 1: rc = -5 [ 5650.655529] LustreError: 189576:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.202.116@tcp: failed processing log, type 4: rc = -110 [ 5680.417407] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5684.164419] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 5690.086633] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5695.251578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5703.396575] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5703.472685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5703.528828] Lustre: lustre-OST0000: deleting orphan objects from 0x0:291 to 0x0:321 [ 5707.265097] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 5708.769966] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5713.896878] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5714.199486] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5714.496326] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5718.511159] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 5724.713728] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5729.113551] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5731.371742] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds in log lustre-OST0000 [ 5744.608863] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5744.624731] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5744.644777] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5745.122657] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5745.141346] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5745.187630] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5747.561157] Lustre: server umount lustre-OST0000 complete [ 5755.360470] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5755.368691] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5755.391769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5755.402709] Lustre: Skipped 2 previous similar messages [ 5757.670919] Lustre: server umount lustre-MDT0000 complete [ 5761.505600] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5761.525182] LustreError: Skipped 1 previous similar message [ 5761.895796] LustreError: 189584:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776198442 with bad export cookie 14547769086669331570 [ 5761.900921] LustreError: 166-1: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5761.903483] LustreError: 189584:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5762.230493] Lustre: server umount lustre-MDT0001 complete [ 5772.318927] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 5774.948630] Key type lgssc unregistered [ 5775.222995] LNet: 193665:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5775.235479] LNet: Removed LNI 192.168.202.116@tcp [ 5775.904119] Key type .llcrypt unregistered [ 5775.905957] Key type ._llcrypt unregistered [ 5796.126208] alg: No test for adler32 (adler32-zlib) [ 5796.875568] Key type ._llcrypt registered [ 5796.878024] Key type .llcrypt registered [ 5796.965177] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 5798.047133] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5798.395711] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 5798.574580] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 5798.578873] LNet: Accept secure, port 988 [ 5800.255270] Key type lgssc registered [ 5801.141594] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5807.941864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5809.850221] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5809.990719] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5815.268516] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5815.620401] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 5820.386608] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5824.359546] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5824.560300] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5829.432499] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 5838.869750] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5844.911530] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5851.323305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5851.705040] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5857.438168] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 5860.948053] Lustre: lustre-OST0000: deleting orphan objects from 0x0:291 to 0x0:353 [ 5865.169849] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5869.710806] Lustre: Failing over lustre-MDT0000 [ 5869.947816] Lustre: server umount lustre-MDT0000 complete [ 5870.564693] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5870.580669] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5870.601643] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5871.080700] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5871.095527] Lustre: Skipped 1 previous similar message [ 5876.195829] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5876.217865] LustreError: Skipped 6 previous similar messages [ 5878.240400] Lustre: 194441:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776198551/real 1776198551] req@00000000f93c5b1f x1862479096328960/t0(0) o400->MGC192.168.202.116@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776198558 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 5878.296778] LustreError: 166-1: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5884.270032] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5884.299913] LustreError: Skipped 8 previous similar messages [ 5891.450642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5895.657981] Lustre: Evicted from MGS (at 192.168.202.116@tcp) after server handle changed from 0x75babfd138b49169 to 0x75babfd138b4953d [ 5895.671778] Lustre: MGC192.168.202.116@tcp: Connection restored to 192.168.202.116@tcp (at 0@lo) [ 5895.998252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5896.110346] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5896.561080] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5901.296658] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 5901.321159] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 5901.366066] Lustre: lustre-OST0000: deleting orphan objects from 0x0:291 to 0x0:385 [ 5901.807829] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 5906.261724] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire in log lustre-MDT0000 [ 5913.940488] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight in log lustre-client [ 5922.277883] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5922.285814] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5922.298659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5924.329963] Lustre: server umount lustre-OST0000 complete [ 5932.008259] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5932.011706] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5932.012251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5932.026231] Lustre: Skipped 1 previous similar message [ 5933.962459] Lustre: server umount lustre-MDT0000 complete [ 5934.112165] Lustre: 194442:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776198607/real 1776198607] req@000000001c50f17e x1862479096345216/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1776198614 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 5934.130748] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5934.136953] LustreError: Skipped 16 previous similar messages [ 5937.181430] LustreError: 197830:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776198617 with bad export cookie 8483303753426179389 [ 5937.185921] LustreError: 166-1: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5937.191618] LustreError: 197830:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5937.559309] Lustre: server umount lustre-MDT0001 complete [ 5947.549197] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 5950.012863] Key type lgssc unregistered [ 5950.289829] LNet: 200239:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5950.309545] LNet: Removed LNI 192.168.202.116@tcp [ 5950.968143] Key type .llcrypt unregistered [ 5950.970987] Key type ._llcrypt unregistered [ 5970.781263] alg: No test for adler32 (adler32-zlib) [ 5971.538452] Key type ._llcrypt registered [ 5971.542380] Key type .llcrypt registered [ 5971.626840] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 5972.752926] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5973.185983] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 5973.388106] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 5973.392722] LNet: Accept secure, port 988 [ 5975.063103] Key type lgssc registered [ 5976.133868] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5983.868848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5985.239969] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5985.285365] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5989.351808] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 5990.382402] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5995.488344] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5998.248311] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5998.729208] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6004.272659] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 6011.797426] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6018.802486] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6025.521506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6025.909500] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6029.783493] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 6036.153798] Lustre: lustre-OST0000: deleting orphan objects from 0x0:387 to 0x0:417 [ 6037.126057] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6039.591599] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 6048.495193] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 6061.208299] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 6075.362338] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6075.369730] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6075.384962] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6077.408986] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6077.410257] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6080.119657] Lustre: server umount lustre-OST0000 complete [ 6085.600585] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6085.614645] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6085.636907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6085.648530] Lustre: Skipped 1 previous similar message [ 6087.650881] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6087.651950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6087.670274] Lustre: Skipped 1 previous similar message [ 6089.456970] Lustre: server umount lustre-MDT0000 complete [ 6092.553095] LustreError: 204441:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776198773 with bad export cookie 6867094158106374390 [ 6092.558276] LustreError: 166-1: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6092.567625] LustreError: 204441:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6092.997271] Lustre: server umount lustre-MDT0001 complete [ 6105.682079] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 6108.138536] Key type lgssc unregistered [ 6108.440342] LNet: 205574:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6108.452162] LNet: Removed LNI 192.168.202.116@tcp [ 6109.059532] Key type .llcrypt unregistered [ 6109.061699] Key type ._llcrypt unregistered [ 6129.103263] alg: No test for adler32 (adler32-zlib) [ 6129.856480] Key type ._llcrypt registered [ 6129.861153] Key type .llcrypt registered [ 6129.986411] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 6131.023687] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6131.399243] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 6131.566553] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 6131.569860] LNet: Accept secure, port 988 [ 6133.240525] Key type lgssc registered [ 6134.316131] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6141.076452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6142.413413] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6142.465513] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6147.552129] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6148.307979] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 6152.673516] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6157.796348] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6158.531967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6159.353613] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6165.531945] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 6175.755302] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6181.211511] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6188.600994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6188.939988] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6193.077318] Lustre: lustre-OST0000: deleting orphan objects from 0x0:387 to 0x0:449 [ 6193.211809] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 6201.516609] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6206.505631] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds in log lustre-OST0000 [ 6215.150239] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds in log lustre-OST0000 [ 6224.542970] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub in log lustre-OST0000 [ 6229.654203] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub in log lustre-OST0000 [ 6236.137023] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6236.146300] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6236.165976] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6239.205483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6239.212408] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6239.226321] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6241.558834] Lustre: server umount lustre-OST0000 complete [ 6246.370123] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6246.377414] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6246.395493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6246.402429] Lustre: Skipped 1 previous similar message [ 6249.440739] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6249.448871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6249.460916] Lustre: Skipped 1 previous similar message [ 6254.562686] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6254.572204] Lustre: Skipped 1 previous similar message [ 6259.167186] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 6259.406251] Lustre: server umount lustre-MDT0000 complete [ 6259.680260] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6259.693248] LustreError: Skipped 1 previous similar message [ 6262.879926] LustreError: 207020:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776198943 with bad export cookie 2663089012657969540 [ 6262.881236] LustreError: 166-1: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6262.891697] LustreError: 207020:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6263.192556] Lustre: server umount lustre-MDT0001 complete [ 6273.814622] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 6276.349460] Key type lgssc unregistered [ 6276.642915] LNet: 211990:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6276.647416] LNet: Removed LNI 192.168.202.116@tcp [ 6277.413986] Key type .llcrypt unregistered [ 6277.416754] Key type ._llcrypt unregistered [ 6298.610930] alg: No test for adler32 (adler32-zlib) [ 6299.384670] Key type ._llcrypt registered [ 6299.389151] Key type .llcrypt registered [ 6299.584418] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 6300.810391] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6301.515845] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 6301.819444] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 6301.833484] LNet: Accept secure, port 988 [ 6303.671519] Key type lgssc registered [ 6305.261112] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6313.978595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6315.506771] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6315.583114] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6320.196908] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 6320.609785] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6325.730697] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6328.343348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6328.616119] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6332.922334] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 6339.166446] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6344.661422] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6351.486094] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6351.703436] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6356.248098] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 6363.693970] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6366.062743] Lustre: lustre-OST0000: deleting orphan objects from 0x0:387 to 0x0:481 [ 6374.969287] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6375.038391] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6375.203089] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6379.287379] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 6386.651249] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6388.365473] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6400.008415] Lustre: Permanently deactivating lustre-OST0001 [ 6400.034685] Lustre: Setting parameter lustre-OST0001-osc.osc.active in log lustre-client [ 6407.755341] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6428.298632] Lustre: Permanently reactivating lustre-OST0001 [ 6428.352244] Lustre: Modifying parameter lustre-OST0001-osc.osc.active in log lustre-client [ 6428.361789] Lustre: Skipped 2 previous similar messages [ 6436.378842] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6436.395061] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6436.414985] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6436.434590] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to (at 0@lo) [ 6436.435325] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:34 to 0x2c0000400:65 [ 6436.466576] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6436.477940] Lustre: lustre-OST0001: deleting orphan objects from 0x0:34 to 0x0:65 [ 6443.493878] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6443.496235] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6443.519130] Lustre: Skipped 2 previous similar messages [ 6443.543908] Lustre: Skipped 1 previous similar message [ 6448.619407] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6448.625328] Lustre: Skipped 1 previous similar message [ 6453.738794] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6453.745379] Lustre: Skipped 1 previous similar message [ 6457.311390] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 6457.352190] Lustre: server umount lustre-OST0001 complete [ 6458.860949] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6458.874979] LustreError: Skipped 1 previous similar message [ 6463.456504] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6463.468971] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6463.496075] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6463.970090] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6463.984315] LustreError: Skipped 1 previous similar message [ 6469.093503] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6469.114388] Lustre: Skipped 2 previous similar messages [ 6474.211655] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6474.229147] LustreError: Skipped 3 previous similar messages [ 6476.255152] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 6476.528781] Lustre: server umount lustre-OST0000 complete [ 6482.402308] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6482.407989] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6482.430596] Lustre: Skipped 1 previous similar message [ 6482.436576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6482.443985] Lustre: Skipped 2 previous similar messages [ 6486.577162] Lustre: server umount lustre-MDT0000 complete [ 6489.868594] LustreError: 216372:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776199170 with bad export cookie 17472099458710075400 [ 6489.873406] LustreError: 166-1: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6489.884893] LustreError: 216372:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6490.170489] Lustre: server umount lustre-MDT0001 complete [ 6500.898468] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 6503.392593] Key type lgssc unregistered [ 6503.748920] LNet: 218640:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6503.762745] LNet: Removed LNI 192.168.202.116@tcp [ 6504.515631] Key type .llcrypt unregistered [ 6504.517703] Key type ._llcrypt unregistered [ 6535.243309] alg: No test for adler32 (adler32-zlib) [ 6536.003714] Key type ._llcrypt registered [ 6536.005254] Key type .llcrypt registered [ 6536.196879] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 6547.680731] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 6548.759353] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6549.037957] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 6549.217337] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 6549.221256] LNet: Accept secure, port 988 [ 6550.871480] Key type lgssc registered [ 6551.947597] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6560.172917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6566.438499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6572.911923] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6580.318608] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6586.454172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6586.509211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6587.734429] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6587.762660] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 6587.842282] Lustre: lustre-MDT0000: new disk, initializing [ 6587.965121] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6588.003948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6592.038870] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 6603.522922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6603.649250] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6603.753900] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6603.802191] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 6603.806828] Lustre: Skipped 1 previous similar message [ 6603.873218] Lustre: lustre-MDT0001: new disk, initializing [ 6603.989064] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6604.021630] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6604.027807] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6608.021258] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 6617.514405] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6622.292967] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6627.770774] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6627.833257] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6627.839454] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 6628.006943] Lustre: lustre-OST0000: new disk, initializing [ 6628.014965] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 6628.091159] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6632.356992] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 6635.538639] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6635.544946] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6641.278197] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6646.326535] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 6646.588893] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6651.065847] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 6651.305928] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6655.982670] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6655.994897] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6656.001278] Lustre: Skipped 1 previous similar message [ 6656.009159] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6656.020229] Lustre: Skipped 1 previous similar message [ 6661.090515] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6661.101635] Lustre: Skipped 1 previous similar message [ 6666.215489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6666.225067] Lustre: Skipped 1 previous similar message [ 6667.231335] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 6667.466298] Lustre: server umount lustre-OST0000 complete [ 6675.939203] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6675.954088] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6675.966233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6677.849351] Lustre: server umount lustre-MDT0000 complete [ 6681.402196] LustreError: 222732:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776199362 with bad export cookie 10172738285301053834 [ 6681.413791] LustreError: 166-1: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6681.643248] Lustre: server umount lustre-MDT0001 complete [ 6688.628035] Lustre: DEBUG MARKER: == conf-sanity test 30a: Big config llog and permanent parameter deletion ========================================================== 16:42:48 (1776199368) [ 6693.948592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6694.355552] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6694.412475] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6698.281682] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 6699.490242] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6704.621871] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6705.743352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6706.022894] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6710.551569] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 6717.725460] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6722.631807] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6727.924383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6728.231129] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6733.094609] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 6740.602875] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6747.519840] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 6750.918774] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 6762.632517] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 6772.689350] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 6772.693774] Lustre: Skipped 1 previous similar message [ 6793.268211] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 6793.275369] Lustre: Skipped 2 previous similar messages [ 6828.414681] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 6828.420563] Lustre: Skipped 4 previous similar messages [ 6898.222985] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 6898.229354] Lustre: Skipped 8 previous similar messages [ 6903.267577] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6903.274608] Lustre: Skipped 2 previous similar messages [ 6903.275960] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6903.282386] Lustre: Skipped 2 previous similar messages [ 6909.146424] Lustre: server umount lustre-OST0000 complete [ 6913.504642] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6913.506219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6913.520212] Lustre: Skipped 1 previous similar message [ 6913.520672] Lustre: Skipped 3 previous similar messages [ 6926.303972] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 6926.422646] Lustre: server umount lustre-MDT0000 complete [ 6928.867294] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6928.887262] LustreError: Skipped 1 previous similar message [ 6929.937720] LustreError: 227267:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776199610 with bad export cookie 10172738285301054681 [ 6929.945276] LustreError: 227267:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6929.946542] LustreError: 166-1: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6939.882814] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 6941.882388] Key type lgssc unregistered [ 6942.086096] LNet: 232154:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6942.099879] LNet: Removed LNI 192.168.202.116@tcp [ 6942.721811] Key type .llcrypt unregistered [ 6942.723880] Key type ._llcrypt unregistered [ 6961.322716] alg: No test for adler32 (adler32-zlib) [ 6962.080368] Key type ._llcrypt registered [ 6962.083302] Key type .llcrypt registered [ 6962.177394] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 6963.331508] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6963.786568] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 6964.010069] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 6964.019195] LNet: Accept secure, port 988 [ 6965.682723] Key type lgssc registered [ 6966.760545] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6973.677398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6974.998178] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6975.045488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6978.562797] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 6980.083871] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6985.184186] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6985.320779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6985.620600] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6988.846458] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 6995.282110] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7000.439128] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7006.964577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7007.321207] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7012.991211] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all 8 [ 7019.595442] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7040.486075] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7040.489716] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7040.530563] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7040.575274] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7040.580925] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.202.136@tcp (at 192.168.202.136@tcp) [ 7042.071500] Lustre: server umount lustre-OST0000 complete [ 7047.138139] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7047.144609] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7047.161956] Lustre: Skipped 1 previous similar message [ 7047.171584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7047.174959] Lustre: Skipped 1 previous similar message [ 7050.720771] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7050.720871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7050.757359] Lustre: Skipped 1 previous similar message [ 7052.464230] Lustre: server umount lustre-MDT0000 complete [ 7055.643139] LustreError: 233596:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776199736 with bad export cookie 3645723822535831635 [ 7055.645962] LustreError: 166-1: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7055.650530] LustreError: 233596:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7055.669554] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_disconnect to node 192.168.202.136@tcp failed: rc = -107 [ 7055.964327] Lustre: server umount lustre-MDT0001 complete [ 7066.391237] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 7068.842134] Key type lgssc unregistered [ 7069.078924] LNet: 237430:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7070.116082] LNet: Removed LNI 192.168.202.116@tcp [ 7070.849245] Key type .llcrypt unregistered [ 7070.853609] Key type ._llcrypt unregistered [ 7090.196107] alg: No test for adler32 (adler32-zlib) [ 7090.950536] Key type ._llcrypt registered [ 7090.952446] Key type .llcrypt registered [ 7091.035560] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 7091.224430] Key type .llcrypt unregistered [ 7091.226181] Key type ._llcrypt unregistered [ 7113.037491] alg: No test for adler32 (adler32-zlib) [ 7113.800764] Key type ._llcrypt registered [ 7113.804282] Key type .llcrypt registered [ 7113.920769] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 7127.605306] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 7128.443129] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7129.357309] Lustre: Lustre: Build Version: 2.15.8_10_g54e838e [ 7129.677899] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 7129.682591] LNet: Accept secure, port 988 [ 7131.375167] Key type lgssc registered [ 7132.786941] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7144.523728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7151.743867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7157.591794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7163.941826] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7165.428492] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 6869 sec ========== 16:50:44 (1776199844) [ 7202.580288] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 7206.312371] Key type lgssc unregistered [ 7206.685962] LNet: 243668:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7206.697045] LNet: Removed LNI 192.168.202.116@tcp [ 7207.965680] Key type .llcrypt unregistered [ 7207.967415] Key type ._llcrypt unregistered