[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 491185619 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001019] APIC: Switch to symmetric I/O mode setup [ 0.003093] x2apic enabled [ 0.005006] Switched APIC routing to physical x2apic. [ 0.006019] kvm-guest: setup PV IPIs [ 0.008663] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009030] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010019] pid_max: default: 32768 minimum: 301 [ 0.011176] LSM: Security Framework initializing [ 0.012089] Yama: becoming mindful. [ 0.014013] SELinux: Initializing. [ 0.015100] *** VALIDATE selinux *** [ 0.024118] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028828] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029196] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031086] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033077] *** VALIDATE tmpfs *** [ 0.034577] *** VALIDATE proc *** [ 0.035351] *** VALIDATE cgroup *** [ 0.036016] *** VALIDATE cgroup2 *** [ 0.038324] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039195] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.041009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.042043] Spectre V2 : User space: Vulnerable [ 0.043012] Speculative Store Bypass: Vulnerable [ 0.046692] debug: unmapping init [mem 0xffffffffb8459000-0xffffffffb8460fff] [ 0.048233] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.049819] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.050031] ... version: 2 [ 0.051020] ... bit width: 48 [ 0.052016] ... generic registers: 4 [ 0.053015] ... value mask: 0000ffffffffffff [ 0.054017] ... max period: 00007fffffffffff [ 0.055015] ... fixed-purpose events: 3 [ 0.055973] ... event mask: 000000070000000f [ 0.056367] rcu: Hierarchical SRCU implementation. [ 0.058717] smp: Bringing up secondary CPUs ... [ 0.059703] x86: Booting SMP configuration: [ 0.060037] .... node #0, CPUs: #1 #2 #3 [ 0.064090] smp: Brought up 1 node, 4 CPUs [ 0.066020] smpboot: Max logical packages: 1 [ 0.067028] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.143000] node 0 deferred pages initialised in 72ms [ 0.146159] devtmpfs: initialized [ 0.148362] x86/mm: Memory block size: 128MB [ 0.151000] gcov: version magic: 0x41383552 [ 0.155363] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.159150] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.161378] pinctrl core: initialized pinctrl subsystem [ 0.163191] [ 0.163695] ************************************************************* [ 0.165017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.167017] ** ** [ 0.169016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.171014] ** ** [ 0.173015] ** This means that this kernel is built to expose internal ** [ 0.175013] ** IOMMU data structures, which may compromise security on ** [ 0.176015] ** your system. ** [ 0.178018] ** ** [ 0.180016] ** If you see this message and you are not debugging the ** [ 0.182017] ** kernel, report this immediately to your vendor! ** [ 0.185019] ** ** [ 0.188018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.190015] ************************************************************* [ 0.192977] NET: Registered protocol family 16 [ 0.195537] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.197077] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.200096] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.203142] cpuidle: using governor menu [ 0.204799] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.206727] PCI: Using configuration type 1 for base access [ 0.209152] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.218082] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.221033] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.225057] cryptd: max_cpu_qlen set to 1000 [ 0.228235] ACPI: Added _OSI(Module Device) [ 0.230024] ACPI: Added _OSI(Processor Device) [ 0.231015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.233015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.237796] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.245605] ACPI: Interpreter enabled [ 0.246000] ACPI: PM: (supports S0 S3 S4 S5) [ 0.248023] ACPI: Using IOAPIC for interrupt routing [ 0.250138] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.254453] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.263440] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.265046] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.267026] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.270144] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.278618] acpiphp: Slot [2] registered [ 0.280158] acpiphp: Slot [5] registered [ 0.282052] acpiphp: Slot [6] registered [ 0.283137] acpiphp: Slot [7] registered [ 0.285159] acpiphp: Slot [8] registered [ 0.286123] acpiphp: Slot [9] registered [ 0.287137] acpiphp: Slot [10] registered [ 0.289135] acpiphp: Slot [3] registered [ 0.291146] acpiphp: Slot [4] registered [ 0.292116] acpiphp: Slot [11] registered [ 0.294153] acpiphp: Slot [12] registered [ 0.296166] acpiphp: Slot [13] registered [ 0.298188] acpiphp: Slot [14] registered [ 0.299119] acpiphp: Slot [15] registered [ 0.301186] acpiphp: Slot [16] registered [ 0.303126] acpiphp: Slot [17] registered [ 0.304132] acpiphp: Slot [18] registered [ 0.306173] acpiphp: Slot [19] registered [ 0.308147] acpiphp: Slot [20] registered [ 0.309140] acpiphp: Slot [21] registered [ 0.311131] acpiphp: Slot [22] registered [ 0.313101] acpiphp: Slot [23] registered [ 0.314126] acpiphp: Slot [24] registered [ 0.316136] acpiphp: Slot [25] registered [ 0.317535] acpiphp: Slot [26] registered [ 0.319136] acpiphp: Slot [27] registered [ 0.321158] acpiphp: Slot [28] registered [ 0.323137] acpiphp: Slot [29] registered [ 0.324194] acpiphp: Slot [30] registered [ 0.326133] acpiphp: Slot [31] registered [ 0.327131] PCI host bridge to bus 0000:00 [ 0.329029] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.331030] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.333031] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.336049] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.339033] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.342039] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.344208] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.347249] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.350817] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.362022] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.367112] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.370032] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.372026] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.375028] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.378702] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.381742] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.384057] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.389049] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.395020] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.411018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.415774] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.422450] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.432019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.441023] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.460023] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.473894] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.481023] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.489021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.507021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.515829] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.522019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.530020] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.544868] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.552936] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.561021] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.568021] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.588022] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.599950] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.606023] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.615021] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.637022] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.651195] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.662017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.672019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.694022] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.708426] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.711437] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.713512] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.717190] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.720275] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.725166] iommu: Default domain type: Passthrough [ 0.727536] SCSI subsystem initialized [ 0.729192] ACPI: bus type USB registered [ 0.731154] usbcore: registered new interface driver usbfs [ 0.733125] usbcore: registered new interface driver hub [ 0.736127] usbcore: registered new device driver usb [ 0.738238] pps_core: LinuxPPS API ver. 1 registered [ 0.741016] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.744082] PTP clock support registered [ 0.746203] EDAC MC: Ver: 3.0.0 [ 0.748136] PCI: Using ACPI for IRQ routing [ 0.750121] NetLabel: Initializing [ 0.751010] NetLabel: domain hash size = 128 [ 0.753016] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.755113] NetLabel: unlabeled traffic allowed by default [ 0.758494] vgaarb: loaded [ 0.760516] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.763021] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.768384] clocksource: Switched to clocksource kvm-clock [ 0.876337] VFS: Disk quotas dquot_6.6.0 [ 0.878071] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.881121] *** VALIDATE ramfs *** [ 0.882394] *** VALIDATE hugetlbfs *** [ 0.884290] pnp: PnP ACPI init [ 0.886826] pnp: PnP ACPI: found 6 devices [ 0.920057] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.923990] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.926570] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.928882] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.931467] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.934346] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.937371] NET: Registered protocol family 2 [ 0.939976] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.945233] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.948989] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.954456] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.958471] TCP: Hash tables configured (established 65536 bind 65536) [ 0.961574] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.964932] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.968271] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.971683] NET: Registered protocol family 1 [ 0.974788] RPC: Registered named UNIX socket transport module. [ 0.976902] RPC: Registered udp transport module. [ 0.978849] RPC: Registered tcp transport module. [ 0.980827] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.983422] NET: Registered protocol family 44 [ 0.985213] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.987467] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.989873] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.992706] PCI: CLS 0 bytes, default 64 [ 0.994522] Unpacking initramfs... [ 2.412363] debug: unmapping init [mem 0xffffa0b97cc54000-0xffffa0b97ffbffff] [ 2.416289] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.418504] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.420923] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.937774] Initialise system trusted keyrings [ 2.940669] Key type blacklist registered [ 2.942876] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.954238] zbud: loaded [ 2.957747] *** VALIDATE nfs *** [ 2.959265] *** VALIDATE nfs4 *** [ 2.961547] pstore: using deflate compression [ 2.965782] Platform Keyring initialized [ 3.078535] NET: Registered protocol family 38 [ 3.080389] Key type asymmetric registered [ 3.081966] Asymmetric key parser 'x509' registered [ 3.083609] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.086262] io scheduler mq-deadline registered [ 3.087708] io scheduler kyber registered [ 3.089485] io scheduler bfq registered [ 3.091162] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.093799] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.096537] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.099028] ACPI: Power Button [PWRF] [ 3.104507] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.111483] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.143385] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.157808] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.179914] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.210765] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.239419] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.244498] Non-volatile memory driver v1.3 [ 3.246483] Linux agpgart interface v0.103 [ 3.280716] virtio_blk virtio1: [vda] 149984 512-byte logical blocks (76.8 MB/73.2 MiB) [ 3.283703] vda: detected capacity change from 0 to 76791808 [ 3.304503] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.307814] vdb: detected capacity change from 0 to 1073741824 [ 3.323397] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.326678] vdc: detected capacity change from 0 to 2621440000 [ 3.341274] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.343772] vdd: detected capacity change from 0 to 2621440000 [ 3.357725] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.359843] vde: detected capacity change from 0 to 4294967296 [ 3.374261] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.376821] vdf: detected capacity change from 0 to 4294967296 [ 3.386080] libphy: Fixed MDIO Bus: probed [ 3.391570] usbcore: registered new interface driver usbserial_generic [ 3.394089] usbserial: USB Serial support registered for generic [ 3.396344] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.400888] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.402431] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.404915] mousedev: PS/2 mouse device common for all mice [ 3.408109] rtc_cmos 00:05: RTC can wake from S4 [ 3.412204] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.412504] rtc_cmos 00:05: registered as rtc0 [ 3.417860] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.420604] intel_pstate: CPU model not supported [ 3.422742] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.423821] hid: raw HID events driver (C) Jiri Kosina [ 3.428675] usbcore: registered new interface driver usbhid [ 3.430933] usbhid: USB HID core driver [ 3.431298] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.433336] drop_monitor: Initializing network drop monitor service [ 3.437949] Initializing XFRM netlink socket [ 3.440430] NET: Registered protocol family 10 [ 3.444059] Segment Routing with IPv6 [ 3.445611] NET: Registered protocol family 17 [ 3.447941] mpls_gso: MPLS GSO support [ 3.453568] RAS: Correctable Errors collector initialized. [ 3.456214] AVX version of gcm_enc/dec engaged. [ 3.458021] AES CTR mode by8 optimization enabled [ 3.538779] sched_clock: Marking stable (3538745233, 0)->(4540307848, -1001562615) [ 3.542348] registered taskstats version 1 [ 3.544554] Loading compiled-in X.509 certificates [ 3.547323] zswap: loaded using pool lzo/zbud [ 3.573186] Key type big_key registered [ 3.587908] Key type encrypted registered [ 3.589775] ima: No TPM chip found, activating TPM-bypass! [ 3.591974] ima: Allocated hash algorithm: sha1 [ 3.593898] ima: No architecture policies found [ 3.596184] evm: Initialising EVM extended attributes: [ 3.598387] evm: security.selinux [ 3.599667] evm: security.ima [ 3.601072] evm: security.capability [ 3.602800] evm: HMAC attrs: 0x1 [ 3.606077] rtc_cmos 00:05: setting system clock to 2026-09-16 16:14:09 UTC (1789575249) [ 3.612583] debug: unmapping init [mem 0xffffffffb9403000-0xffffffffb95fffff] [ 3.616213] debug: unmapping init [mem 0xffffffffb8182000-0xffffffffb8458fff] [ 3.624962] Write protecting the kernel read-only data: 28672k [ 3.628687] debug: unmapping init [mem 0xffffffffb6803000-0xffffffffb69fffff] [ 3.631874] debug: unmapping init [mem 0xffffffffb7114000-0xffffffffb71fffff] [ 3.671405] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.680882] systemd[1]: Detected virtualization kvm. [ 3.683042] systemd[1]: Detected architecture x86-64. [ 3.685473] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.711618] systemd[1]: No hostname configured. [ 3.713750] systemd[1]: Set hostname to . [ 3.716260] random: systemd: uninitialized urandom read (16 bytes read) [ 3.719181] systemd[1]: Initializing machine ID from random generator. [ 3.764464] random: ln: uninitialized urandom read (6 bytes read) [ 3.854357] random: systemd: uninitialized urandom read (16 bytes read) [ 3.856823] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.862873] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.866775] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. Starting Setup Virtual Console... [ OK ] Reached target Swap. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Journal Service. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.526909] device-mapper: uevent: version 1.0.3 [ 4.529194] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.313485] random: fast init done [ 5.337477] virtio_net virtio0 ens2: renamed from eth0 [ 5.461514] scsi host0: ata_piix [ 5.491102] scsi host1: ata_piix [ 5.492902] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.495389] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.166782] dracut-initqueue[581]: RTNETLINK answers: File exists [ 10.109415] random: crng init done [ 10.111849] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.659698] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.861298] printk: systemd: 21 output lines suppressed due to ratelimiting [ 12.153118] SELinux: Disabled at runtime. [ 12.218814] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.230241] systemd[1]: Detected virtualization kvm. [ 12.232358] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.798379] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.801995] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.807113] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.811205] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.814812] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.826038] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.836281] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target rpc_pipefs.target. Starting Create list of required st…ce nodes for the current kernel... Mounting Kernel Debug File System... [ OK ] Listening on udev Control Socket. [ 12.867728] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Apply Kernel Variables... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting POSIX Message Queue File System... [ OK ] Created slice system-getty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Remount Root and Kernel File Systems... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-sshd\x2dkeygen.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Mounting Huge Pages File System... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 13.318869] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.611362] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.692855] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.823310] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.833518] EDAC sbridge: Ver: 1.1.2 [ 15.711278] Key type dns_resolver registered [ 16.017538] NFS: Registering the id_resolver key type [ 16.019805] Key type id_resolver registered [ 16.021332] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg434-server login: [ 67.881457] libcfs: loading out-of-tree module taints kernel. [ 67.906043] Key type ._llcrypt registered [ 67.907904] Key type .llcrypt registered [ 68.054100] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_hostid [ 87.431343] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 88.898984] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 88.914353] alg: No test for adler32 (adler32-zlib) [ 90.551922] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 91.676334] LNet: Added LNI 192.168.204.134@tcp [8/256/0/180] [ 93.447489] Key type lgssc registered [ 95.731410] Lustre: Echo OBD driver; http://www.lustre.org/ [ 117.457300] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 166.374357] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 181.886135] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 181.937814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 183.413726] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 183.468862] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 183.661192] Lustre: lustre-MDT0000: new disk, initializing [ 183.808222] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 183.847799] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 188.369795] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 203.135816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 203.346387] Lustre: 6517:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 203.443510] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 203.447513] Lustre: Skipped 1 previous similar message [ 203.587926] Lustre: lustre-MDT0001: new disk, initializing [ 203.672571] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 203.699837] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 203.714923] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 208.118905] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 212.925228] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 222.892526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 223.140705] Lustre: lustre-OST0000: new disk, initializing [ 223.144403] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 223.149949] Lustre: 8455:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 223.219383] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 229.898477] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 232.982900] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 232.987630] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 233.045608] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 247.584441] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 247.687898] Lustre: lustre-OST0001: new disk, initializing [ 247.691995] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 247.709217] Lustre: 9527:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 247.809588] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 252.530641] hrtimer: interrupt took 4047179 ns [ 253.534434] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 253.560458] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 253.714298] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 255.748277] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 271.467734] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 279.540501] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 287.094765] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing check_logdir /tmp/testlogs/ [ 292.445481] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing yml_node [ 297.739965] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 300.569853] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 303.152771] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 305.270800] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Sep 16 12:19:09 EDT 2026 [ 322.459969] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 324.038174] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 340.967831] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 340.969175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 340.978586] Lustre: Skipped 2 previous similar messages [ 340.995834] Lustre: Skipped 3 previous similar messages [ 346.083974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 346.099047] Lustre: Skipped 3 previous similar messages [ 351.209228] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 351.220061] Lustre: Skipped 3 previous similar messages [ 354.271149] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 354.556445] Lustre: server umount lustre-MDT0000 complete [ 358.476545] LustreError: 6509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789575604 with bad export cookie 11696134885633557738 [ 358.480824] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 358.495255] LustreError: 6509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 361.440713] LustreError: 10267:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 361.441797] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 361.448654] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 361.464556] LustreError: 10267:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 361.484704] Lustre: Skipped 2 previous similar messages [ 363.488267] LustreError: 6524:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 363.501414] LustreError: 6524:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 364.897632] Lustre: server umount lustre-MDT0001 complete [ 368.546829] Lustre: server umount lustre-OST0000 complete [ 372.459445] Lustre: server umount lustre-OST0001 complete [ 380.021265] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_hostid [ 389.532696] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 429.800679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 430.160200] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 430.183112] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 430.241661] Lustre: lustre-MDT0000: new disk, initializing [ 430.400957] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 430.435574] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 436.164696] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 449.639380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 449.781838] Lustre: 15080:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 449.834748] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 449.837438] Lustre: Skipped 1 previous similar message [ 449.938273] Lustre: lustre-MDT0001: new disk, initializing [ 450.041296] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 450.092843] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 450.107119] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 454.844391] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 465.457868] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 471.954356] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 478.950220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 479.279956] Lustre: lustre-OST0000: new disk, initializing [ 479.289752] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 479.295468] Lustre: 17050:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 479.408711] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 481.235681] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 481.253126] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 481.424803] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 486.360796] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 496.885878] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 502.799345] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 503.158803] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 508.621268] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 508.840756] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 511.970615] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 511.976475] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 511.979498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 511.984278] Lustre: Skipped 1 previous similar message [ 511.996078] Lustre: Skipped 4 previous similar messages [ 517.049298] Lustre: server umount lustre-OST0000 complete [ 521.698103] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 521.718118] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 521.757561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 527.672918] Lustre: server umount lustre-MDT0000 complete [ 531.428677] LustreError: 16684:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 531.446070] LustreError: 16684:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 532.406299] LustreError: 15877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789575778 with bad export cookie 11696134885633559271 [ 532.408641] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 532.423657] LustreError: 15877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 532.870723] Lustre: server umount lustre-MDT0001 complete [ 537.852923] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 12:23:02 (1789575782) [ 545.904179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 546.735458] LustreError: 19315:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 546.826765] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 551.904644] LustreError: 19316:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 553.031133] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 562.151331] LustreError: 19315:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 562.186051] LustreError: 19315:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 563.779966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 569.576099] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 577.984359] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 584.452303] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 593.500251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 600.915364] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 609.780731] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 630.241098] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 630.253472] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 630.266425] Lustre: Skipped 1 previous similar message [ 630.271641] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 630.277164] Lustre: Skipped 4 previous similar messages [ 632.303462] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 633.892472] Lustre: server umount lustre-OST0000 complete [ 640.994924] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 641.003740] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 641.015346] Lustre: Skipped 1 previous similar message [ 644.080954] Lustre: server umount lustre-MDT0000 complete [ 647.648639] LustreError: 19310:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 647.668619] LustreError: 19310:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 648.001515] LustreError: 19298:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789575893 with bad export cookie 11696134885633560153 [ 648.019982] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 648.308750] Lustre: server umount lustre-MDT0001 complete [ 660.431555] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [ 663.696516] Key type lgssc unregistered [ 664.027699] LNet: 22870:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 664.036176] LNetError: 22870:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 664.058717] LNet: Removed LNI 192.168.204.134@tcp [ 665.238737] Key type .llcrypt unregistered [ 665.240493] Key type ._llcrypt unregistered [ 694.616565] Key type ._llcrypt registered [ 694.620101] Key type .llcrypt registered [ 694.741807] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 696.096583] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 696.115619] alg: No test for adler32 (adler32-zlib) [ 697.234647] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 697.490409] LNet: Added LNI 192.168.204.134@tcp [8/256/0/180] [ 699.208513] Key type lgssc registered [ 700.371859] Lustre: Echo OBD driver; http://www.lustre.org/ [ 709.809123] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 709.850416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 711.570742] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 717.176440] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 721.906114] LustreError: 24281:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 721.929554] LustreError: 24281:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 726.717202] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 727.015179] LustreError: 24282:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 727.398763] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 732.504094] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 741.380602] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 748.463389] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 756.353943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 756.926621] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 765.249527] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 774.321599] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 781.508154] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 793.059710] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 793.072195] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 793.102790] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 794.099836] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 794.114592] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 794.146666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 794.752032] Lustre: server umount lustre-OST0000 complete [ 802.278091] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 802.284470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 802.302178] Lustre: Skipped 1 previous similar message [ 802.318414] Lustre: Skipped 1 previous similar message [ 805.702719] Lustre: server umount lustre-MDT0000 complete [ 807.396707] LustreError: 24281:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 807.428200] LustreError: 24281:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 810.476049] LustreError: 24263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789576056 with bad export cookie 15282718657883251780 [ 810.484560] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 810.494918] LustreError: 24263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 811.235868] Lustre: server umount lustre-MDT0001 complete [ 824.541349] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [ 828.142809] Key type lgssc unregistered [ 828.538593] LNet: 28257:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 828.557414] LNetError: 28257:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 828.585696] LNet: Removed LNI 192.168.204.134@tcp [ 829.617985] Key type .llcrypt unregistered [ 829.619839] Key type ._llcrypt unregistered [ 854.672208] Key type ._llcrypt registered [ 854.678991] Key type .llcrypt registered [ 854.819553] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 856.561567] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 856.580776] alg: No test for adler32 (adler32-zlib) [ 857.710564] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 858.052283] LNet: Added LNI 192.168.204.134@tcp [8/256/0/180] [ 859.847490] Key type lgssc registered [ 861.510072] Lustre: Echo OBD driver; http://www.lustre.org/ [ 870.793532] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 870.853862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 872.705471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 877.381952] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 882.151088] LustreError: 29670:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 882.175753] LustreError: 29670:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 886.949257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 887.338726] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 893.209752] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 903.396918] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 911.244709] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 925.628228] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 926.060315] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 932.498991] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 939.994549] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 942.541258] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 952.820762] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 952.828491] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 952.859495] Lustre: Skipped 1 previous similar message [ 952.868503] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 952.875551] Lustre: Skipped 1 previous similar message [ 957.934340] LustreError: 31839:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 957.961714] LustreError: 31839:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 958.136918] Lustre: server umount lustre-OST0000 complete [ 963.052514] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 963.057918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 963.075820] Lustre: Skipped 1 previous similar message [ 968.164832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 968.175734] Lustre: Skipped 2 previous similar messages [ 968.532210] Lustre: server umount lustre-MDT0000 complete [ 973.284674] LustreError: 29670:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 973.306956] LustreError: 29670:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 973.470449] LustreError: 31860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789576219 with bad export cookie 254704149598771625 [ 973.475303] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 973.492952] LustreError: 31860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 973.994495] Lustre: server umount lustre-MDT0001 complete [ 986.849547] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [ 990.158760] Key type lgssc unregistered [ 990.504904] LNet: 33550:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 990.512845] LNetError: 33550:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 990.539869] LNet: Removed LNI 192.168.204.134@tcp [ 991.769108] Key type .llcrypt unregistered [ 991.772933] Key type ._llcrypt unregistered [ 1014.027436] Key type ._llcrypt registered [ 1014.029880] Key type .llcrypt registered [ 1014.172950] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 1015.741971] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1015.894100] alg: No test for adler32 (adler32-zlib) [ 1016.999585] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 1017.352305] LNet: Added LNI 192.168.204.134@tcp [8/256/0/180] [ 1019.079254] Key type lgssc registered [ 1020.585829] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1030.487839] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1030.538374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1032.611544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1039.423292] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 1042.915425] LustreError: 34960:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1042.928416] LustreError: 34960:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1048.033204] LustreError: 34959:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1049.417333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1049.953894] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1055.554980] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 1063.832764] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1069.452572] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1076.756619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1077.095959] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1078.160694] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1083.078701] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 1090.460373] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1103.843306] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1103.856659] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1103.872905] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1104.865737] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1104.881244] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1104.896963] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1108.935707] Lustre: server umount lustre-OST0000 complete [ 1115.106290] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1115.136974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1115.152291] Lustre: Skipped 2 previous similar messages [ 1120.224780] LustreError: 35688:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1120.230178] Lustre: server umount lustre-MDT0000 complete [ 1120.240181] LustreError: 35688:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1124.035146] LustreError: 34938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789576369 with bad export cookie 2441077547304060346 [ 1124.069130] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1124.454930] Lustre: server umount lustre-MDT0001 complete [ 1135.223683] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [ 1137.931799] Key type lgssc unregistered [ 1138.262607] LNet: 38505:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1138.276071] LNetError: 38505:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1138.315371] LNet: Removed LNI 192.168.204.134@tcp [ 1139.147483] Key type .llcrypt unregistered [ 1139.170241] Key type ._llcrypt unregistered [ 1166.765975] Key type ._llcrypt registered [ 1166.769372] Key type .llcrypt registered [ 1166.895270] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 1168.053267] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1168.072820] alg: No test for adler32 (adler32-zlib) [ 1169.160906] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 1169.487916] LNet: Added LNI 192.168.204.134@tcp [8/256/0/180] [ 1171.255309] Key type lgssc registered [ 1172.366953] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1179.492619] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1179.509555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1180.840961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1184.940379] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 1190.372023] LustreError: 39917:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1190.386609] LustreError: 39917:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1193.333494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1193.715313] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1197.924376] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 1205.188142] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1211.330788] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1217.488205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1217.931426] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1224.334347] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 1225.208585] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1233.001693] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1240.546159] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1240.553234] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1240.599937] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1245.669673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1245.680701] Lustre: Skipped 3 previous similar messages [ 1250.788327] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1250.797642] Lustre: Skipped 2 previous similar messages [ 1254.367184] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1254.572672] Lustre: server umount lustre-OST0000 complete [ 1275.873107] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1275.889970] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1275.908205] Lustre: Skipped 1 previous similar message [ 1275.915928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1275.921416] Lustre: Skipped 1 previous similar message [ 1280.482903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1280.493150] Lustre: Skipped 3 previous similar messages [ 1281.974300] Lustre: server umount lustre-MDT0000 complete [ 1285.608334] LustreError: 40646:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1285.639706] LustreError: 40646:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1285.830263] LustreError: 39899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789576531 with bad export cookie 2663348574276050665 [ 1285.835036] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1285.842782] LustreError: 39899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1292.380779] Lustre: server umount lustre-MDT0001 complete [ 1303.400495] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [ 1306.301526] Key type lgssc unregistered [ 1306.588931] LNet: 43628:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1306.600825] LNetError: 43628:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1306.623694] LNet: Removed LNI 192.168.204.134@tcp [ 1307.402898] Key type .llcrypt unregistered [ 1307.405511] Key type ._llcrypt unregistered [ 1330.464295] Key type ._llcrypt registered [ 1330.468549] Key type .llcrypt registered [ 1330.633182] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 1331.560507] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1331.585610] alg: No test for adler32 (adler32-zlib) [ 1332.698043] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 1332.937728] LNet: Added LNI 192.168.204.134@tcp [8/256/0/180] [ 1334.615221] Key type lgssc registered [ 1335.574153] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1343.877617] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1343.891827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1345.727519] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1350.618277] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 1355.234220] LustreError: 45039:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1355.257588] LustreError: 45039:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1360.358687] LustreError: 45040:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1362.008136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1362.401258] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1367.394106] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 1375.822707] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1382.595951] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1389.642448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1390.028818] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1391.096834] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1396.850641] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 1404.820809] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1417.537673] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.34@tcp (stopping) [ 1417.702703] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1417.716957] Lustre: Skipped 2 previous similar messages [ 1420.843103] Lustre: server umount lustre-MDT0000 complete [ 1422.664707] LustreError: 46241:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1424.776106] LustreError: 45020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789576670 with bad export cookie 11189667264686159693 [ 1424.787652] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1424.790563] LustreError: 45020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1427.758335] LustreError: 45035:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1427.772662] LustreError: 45035:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1427.794742] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.34@tcp (stopping) [ 1427.810358] Lustre: Skipped 3 previous similar messages [ 1427.936792] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1429.986084] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1429.995403] Lustre: Skipped 1 previous similar message [ 1432.883689] LustreError: 45035:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1432.884094] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.34@tcp (stopping) [ 1432.899557] LustreError: 45035:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1437.994242] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.34@tcp (stopping) [ 1438.001086] Lustre: Skipped 1 previous similar message [ 1439.200069] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1439.429657] Lustre: server umount lustre-MDT0001 complete [ 1454.157791] Lustre: server umount lustre-OST0000 complete [ 1474.061564] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [ 1476.824744] Key type lgssc unregistered [ 1477.166942] LNet: 48901:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1477.171071] LNetError: 48901:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1477.190349] LNet: Removed LNI 192.168.204.134@tcp [ 1478.089753] Key type .llcrypt unregistered [ 1478.094675] Key type ._llcrypt unregistered [ 1505.606414] Key type ._llcrypt registered [ 1505.609265] Key type .llcrypt registered [ 1505.752239] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 1506.944980] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1507.171570] alg: No test for adler32 (adler32-zlib) [ 1508.363819] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 1508.676798] LNet: Added LNI 192.168.204.134@tcp [8/256/0/180] [ 1510.399299] Key type lgssc registered [ 1511.466469] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1519.073418] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1519.098512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1534.688169] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1539.807497] LustreError: 50284:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.134@tcp: failed processing log, type 4: rc = -110 [ 1566.495268] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1573.195197] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 1581.143375] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1750.036859] Lustre: server umount lustre-OST0000 complete [ 1768.325446] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [ 1771.298352] Key type lgssc unregistered [ 1771.729899] LNet: 51958:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1771.739848] LNetError: 51958:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1771.759359] LNet: Removed LNI 192.168.204.134@tcp [ 1772.701980] Key type .llcrypt unregistered [ 1772.708477] Key type ._llcrypt unregistered [ 1798.417136] Key type ._llcrypt registered [ 1798.419391] Key type .llcrypt registered [ 1798.582671] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 1800.102040] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1800.136421] alg: No test for adler32 (adler32-zlib) [ 1801.287602] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 1801.561581] LNet: Added LNI 192.168.204.134@tcp [8/256/0/180] [ 1803.207278] Key type lgssc registered [ 1804.474477] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1814.499727] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1814.548196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1816.458966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1821.908989] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 1826.807049] LustreError: 53370:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1826.817487] LustreError: 53370:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1831.904746] LustreError: 53371:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1832.110703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1832.512909] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1836.457964] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 1845.692693] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1852.029815] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1859.458758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1859.879864] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1861.939223] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1866.215065] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 1874.167410] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1877.483753] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1877.488139] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1877.507474] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1882.596550] LustreError: 55191:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1882.751994] Lustre: server umount lustre-OST0000 complete [ 1887.713320] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1887.721473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1887.727909] Lustre: Skipped 1 previous similar message [ 1887.738949] Lustre: Skipped 1 previous similar message [ 1892.852776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1892.866225] Lustre: Skipped 2 previous similar messages [ 1893.349029] Lustre: server umount lustre-MDT0000 complete [ 1897.933709] LustreError: 53352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789577143 with bad export cookie 11488958027943076721 [ 1897.934969] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1897.948691] LustreError: 53352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1898.511729] Lustre: server umount lustre-MDT0001 complete [ 1911.363229] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [ 1914.104196] Key type lgssc unregistered [ 1914.383149] LNet: 56902:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1914.390953] LNetError: 56902:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1914.410468] LNet: Removed LNI 192.168.204.134@tcp [ 1915.301727] Key type .llcrypt unregistered [ 1915.303538] Key type ._llcrypt unregistered [ 1937.366474] Key type ._llcrypt registered [ 1937.368632] Key type .llcrypt registered [ 1937.437803] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 1938.069756] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1938.115538] alg: No test for adler32 (adler32-zlib) [ 1939.287253] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 1939.578871] LNet: Added LNI 192.168.204.134@tcp [8/256/0/180] [ 1941.263197] Key type lgssc registered [ 1942.136651] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1950.184288] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1950.214039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1965.855403] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1970.977571] LustreError: 58284:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.134@tcp: failed processing log, type 4: rc = -110 [ 1997.663265] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2004.569933] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 2013.102939] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2022.312320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2023.246458] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2023.356813] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2028.514684] LustreError: 58309:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2028.552436] LustreError: 58309:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2028.940699] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 2032.614559] LustreError: 59321:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2032.659021] LustreError: 59321:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2037.733156] LustreError: 58309:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2037.748306] LustreError: 58309:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2039.188739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2039.751445] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2045.523577] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 2055.113525] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2061.873226] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2064.354828] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2064.363400] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2064.376494] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2065.381786] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2065.383695] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2065.408354] Lustre: Skipped 1 previous similar message [ 2070.552961] Lustre: server umount lustre-OST0000 complete [ 2096.101249] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2096.105915] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2096.118210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2096.122440] Lustre: Skipped 1 previous similar message [ 2101.217883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2101.228407] Lustre: Skipped 2 previous similar messages [ 2105.313376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2105.323585] Lustre: Skipped 1 previous similar message [ 2107.359678] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2107.847330] Lustre: server umount lustre-MDT0000 complete [ 2110.434311] LustreError: 59324:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2110.450465] LustreError: 59324:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2112.409611] LustreError: 58292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789577358 with bad export cookie 16557947357040306742 [ 2112.430586] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2118.978439] Lustre: server umount lustre-MDT0001 complete [ 2132.341853] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [ 2135.556493] Key type lgssc unregistered [ 2135.966316] LNet: 62030:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2135.980566] LNetError: 62030:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2136.006635] LNet: Removed LNI 192.168.204.134@tcp [ 2137.155293] Key type .llcrypt unregistered [ 2137.157128] Key type ._llcrypt unregistered [ 2163.321280] Key type ._llcrypt registered [ 2163.325540] Key type .llcrypt registered [ 2163.443470] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 2164.503161] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2164.541666] alg: No test for adler32 (adler32-zlib) [ 2165.657455] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 2165.972501] LNet: Added LNI 192.168.204.134@tcp [8/256/0/180] [ 2167.681677] Key type lgssc registered [ 2168.942695] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2177.605119] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2177.657910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2179.309377] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2184.186314] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 2189.799182] LustreError: 63443:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2189.825404] LustreError: 63443:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2194.006837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2194.552298] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2199.868686] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 2208.196756] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2215.340091] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2225.026868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2225.612535] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2233.579968] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 2244.894582] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2249.159392] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2261.800500] Lustre: server umount lustre-OST0000 complete [ 2269.668394] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2269.669862] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2269.682518] Lustre: Skipped 1 previous similar message [ 2269.698690] Lustre: Skipped 1 previous similar message [ 2274.785888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2274.791666] Lustre: Skipped 1 previous similar message [ 2276.892768] Lustre: server umount lustre-MDT0000 complete [ 2279.918913] LustreError: 64172:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2279.954720] LustreError: 64172:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2280.143685] Lustre: 62752:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789577510/real 1789577510] req@00000000fad6f2b1 x1876507923347712/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789577526 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2280.208177] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2281.258985] Lustre: 62754:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789577511/real 1789577511] req@00000000b135935b x1876507923348096/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789577527 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2282.304377] LustreError: 63424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789577528 with bad export cookie 846894702664318162 [ 2282.314975] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2282.320628] LustreError: 63424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2282.798124] Lustre: server umount lustre-MDT0001 complete [ 2299.799891] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [ 2302.921394] Key type lgssc unregistered [ 2303.357878] LNet: 66998:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2303.373822] LNetError: 66998:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2303.413626] LNet: Removed LNI 192.168.204.134@tcp [ 2304.579673] Key type .llcrypt unregistered [ 2304.581645] Key type ._llcrypt unregistered [ 2338.080672] Key type ._llcrypt registered [ 2338.084638] Key type .llcrypt registered [ 2338.241851] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 2339.674441] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2339.700039] alg: No test for adler32 (adler32-zlib) [ 2340.786508] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 2341.041150] LNet: Added LNI 192.168.204.134@tcp [8/256/0/180] [ 2342.735376] Key type lgssc registered [ 2344.262188] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2356.085782] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 12:53:20 (1789577600) [ 2364.139847] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2364.166381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2365.711636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2370.452346] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 2375.139973] LustreError: 68700:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2375.163264] LustreError: 68700:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2379.926153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2380.363731] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2385.148613] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 2393.623390] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2401.523784] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2409.191426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2409.514370] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2410.567050] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2416.057858] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 2424.347604] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2431.884282] Lustre: Failing over lustre-MDT0000 [ 2432.248887] Lustre: server umount lustre-MDT0000 complete [ 2432.337345] LustreError: 68697:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2436.069761] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2436.086829] LustreError: 70847:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2436.091115] Lustre: Skipped 1 previous similar message [ 2441.195237] LustreError: 68696:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2441.222466] LustreError: 68696:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 2442.825661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2442.982120] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2443.143982] Lustre: *** cfs_fail_loc=135, val=0*** [ 2443.252563] LustreError: 71789:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2443.257155] LustreError: 71789:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.134@tcp: cfg command failed: rc = -2 [ 2443.265552] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2443.265552] [ 2443.278039] LustreError: MGC192.168.204.134@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2443.290531] LustreError: 71778:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2443.299186] LustreError: 71778:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 2443.310804] LustreError: 71778:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 2443.315846] LustreError: 71778:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2443.343471] Lustre: server umount lustre-MDT0000 complete [ 2443.345368] LustreError: 71778:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2446.305211] LustreError: 68701:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2446.330677] LustreError: 68701:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2451.553831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2451.790326] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2452.182113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2452.266452] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2452.818535] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2457.603536] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2457.666385] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2457.704792] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2457.996625] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 2471.908095] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2471.920933] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2471.935582] Lustre: Skipped 1 previous similar message [ 2471.948838] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2472.932735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2472.950967] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2476.260349] Lustre: server umount lustre-OST0000 complete [ 2483.168654] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2483.174310] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2483.187716] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2483.190238] Lustre: Skipped 1 previous similar message [ 2488.291203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2488.298277] Lustre: Skipped 1 previous similar message [ 2489.117512] Lustre: server umount lustre-MDT0000 complete [ 2493.408420] LustreError: 68700:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2493.450816] LustreError: 68700:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 2494.489173] LustreError: 68682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789577740 with bad export cookie 12070843025883720453 [ 2494.491104] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2494.499381] LustreError: 68682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2494.873439] Lustre: server umount lustre-MDT0001 complete [ 2506.640723] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [ 2509.959783] Key type lgssc unregistered [ 2510.319482] LNet: 73569:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2510.367229] LNetError: 73569:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2510.398862] LNet: Removed LNI 192.168.204.134@tcp [ 2511.477739] Key type .llcrypt unregistered [ 2511.479334] Key type ._llcrypt unregistered [ 2543.056380] Key type ._llcrypt registered [ 2543.058896] Key type .llcrypt registered [ 2543.204481] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 2544.623948] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2544.653063] alg: No test for adler32 (adler32-zlib) [ 2545.809336] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 2546.150910] LNet: Added LNI 192.168.204.134@tcp [8/256/0/180] [ 2547.871940] Key type lgssc registered [ 2549.333634] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2558.418590] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2558.432461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2560.077512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2565.448743] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 2569.193679] LustreError: 74980:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2569.215753] LustreError: 74980:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2574.309285] LustreError: 74979:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2575.265875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2575.590323] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2580.856525] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 2589.502638] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2596.453403] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2605.974076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2606.236488] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2613.111477] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 2622.455336] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2629.815543] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2638.609167] Lustre: Failing over lustre-MDT0000 [ 2638.684565] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.34@tcp (stopping) [ 2638.996582] Lustre: server umount lustre-MDT0000 complete [ 2640.355688] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2640.363488] LustreError: 74975:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2640.374684] Lustre: Skipped 2 previous similar messages [ 2640.415505] LustreError: 74975:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2643.756634] LustreError: 74974:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2648.890229] LustreError: 76567:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2648.935578] LustreError: 76567:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2651.146206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2651.268493] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2651.400603] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2651.566374] LustreError: 78093:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2651.575731] LustreError: 78093:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.134@tcp: cfg command failed: rc = -9 [ 2651.586834] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2651.586834] [ 2651.597475] LustreError: MGC192.168.204.134@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2651.625895] LustreError: 78081:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2651.651102] LustreError: 78081:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -9 [ 2651.672733] LustreError: 78081:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 2651.679927] LustreError: 78081:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2651.718284] Lustre: server umount lustre-MDT0000 complete [ 2651.726731] LustreError: 78081:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2659.120306] LustreError: 76567:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2659.169383] LustreError: 76567:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 2661.656099] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2662.064138] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2662.084716] LustreError: 74971:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2662.504753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2662.582895] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2664.252553] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2667.514825] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2667.543340] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2667.592287] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2668.128876] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 2677.216253] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2677.224231] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2677.242238] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2681.769852] Lustre: server umount lustre-OST0000 complete [ 2687.973182] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2687.983290] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2687.990583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2688.013533] Lustre: Skipped 2 previous similar messages [ 2688.016428] Lustre: Skipped 1 previous similar message [ 2692.985634] Lustre: server umount lustre-MDT0000 complete [ 2693.090817] LustreError: 74980:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2693.117638] LustreError: 74980:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 2697.541480] LustreError: 76823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789577943 with bad export cookie 411127563789217892 [ 2697.546796] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2697.564883] LustreError: 76823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2698.122742] Lustre: server umount lustre-MDT0001 complete [ 2710.980778] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [ 2715.074330] Key type lgssc unregistered [ 2715.493692] LNet: 79875:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2715.519286] LNetError: 79875:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2715.567803] LNet: Removed LNI 192.168.204.134@tcp [ 2717.054847] Key type .llcrypt unregistered [ 2717.057782] Key type ._llcrypt unregistered [ 2734.320378] Key type ._llcrypt registered [ 2734.325409] Key type .llcrypt registered [ 2734.430159] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [ 2734.726621] Key type .llcrypt unregistered [ 2734.731050] Key type ._llcrypt unregistered [ 2764.506518] Key type ._llcrypt registered [ 2764.509910] Key type .llcrypt registered [ 2764.672566] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 2766.530650] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2766.669815] alg: No test for adler32 (adler32-zlib) [ 2767.819632] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 2768.383790] LNet: Added LNI 192.168.204.134@tcp [8/256/0/180] [ 2770.159407] Key type lgssc registered [ 2771.988700] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2781.685151] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2781.742395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2783.528482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2789.044370] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 2793.954761] LustreError: 81772:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2793.970846] LustreError: 81772:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2799.081135] LustreError: 81771:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2799.147126] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2799.692085] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2804.441477] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 2812.600916] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2819.530112] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2827.495816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2828.023201] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2829.047617] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2835.381754] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 2844.120365] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2854.886156] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2854.896344] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2854.934888] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2855.912774] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2855.920600] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2855.927240] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2860.039493] Lustre: server umount lustre-OST0000 complete [ 2866.145796] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2866.155717] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2866.167450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2866.181535] Lustre: Skipped 1 previous similar message [ 2866.190699] Lustre: Skipped 2 previous similar messages [ 2870.682219] Lustre: server umount lustre-MDT0000 complete [ 2871.275530] LustreError: 81771:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2871.303325] LustreError: 81771:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2875.125868] LustreError: 81751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789578121 with bad export cookie 3500463099975641365 [ 2875.144189] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2875.145503] LustreError: 81751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2875.400734] Lustre: server umount lustre-MDT0001 complete [ 2888.556505] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [ 2891.751870] Key type lgssc unregistered [ 2892.062857] LNet: 85317:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2892.068103] LNetError: 85317:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2892.091677] LNet: Removed LNI 192.168.204.134@tcp [ 2893.129444] Key type .llcrypt unregistered [ 2893.136478] Key type ._llcrypt unregistered [ 2923.611494] Key type ._llcrypt registered [ 2923.613655] Key type .llcrypt registered [ 2923.760644] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 2925.345464] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2925.462258] alg: No test for adler32 (adler32-zlib) [ 2926.609301] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 2926.986218] LNet: Added LNI 192.168.204.134@tcp [8/256/0/180] [ 2928.631487] Key type lgssc registered [ 2929.612695] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2938.325241] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2938.383104] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2940.317920] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2945.593700] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 2950.626966] LustreError: 86728:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2950.640748] LustreError: 86728:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2955.777476] LustreError: 86729:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2955.824383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2956.303293] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2961.903536] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 2971.185543] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2978.719616] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2987.499532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2987.836477] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2994.702223] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 3003.789537] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3010.376436] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 3020.769913] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3020.772336] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3020.791738] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3020.809382] Lustre: Skipped 1 previous similar message [ 3025.909753] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3025.917195] Lustre: Skipped 1 previous similar message [ 3026.579165] Lustre: server umount lustre-OST0000 complete [ 3031.018144] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3031.026927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3031.031159] Lustre: Skipped 2 previous similar messages [ 3031.050758] Lustre: Skipped 1 previous similar message [ 3036.140253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3036.147666] Lustre: Skipped 1 previous similar message [ 3041.262204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3041.280612] Lustre: Skipped 1 previous similar message [ 3045.343910] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3045.696170] Lustre: server umount lustre-MDT0000 complete [ 3046.383117] LustreError: 86728:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3046.433612] LustreError: 86728:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3050.466577] LustreError: 86723:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3050.480508] LustreError: 86723:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3051.684967] LustreError: 86708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789578297 with bad export cookie 5396666409008522322 [ 3051.699180] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3051.703877] LustreError: 86708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3052.324945] Lustre: server umount lustre-MDT0001 complete [ 3067.082696] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [ 3070.250930] Key type lgssc unregistered [ 3070.637556] LNet: 90288:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3070.648770] LNetError: 90288:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3070.702130] LNet: Removed LNI 192.168.204.134@tcp [ 3071.770649] Key type .llcrypt unregistered [ 3071.772279] Key type ._llcrypt unregistered [ 3102.985274] Key type ._llcrypt registered [ 3102.990691] Key type .llcrypt registered [ 3103.133678] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 3104.394229] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3104.458204] alg: No test for adler32 (adler32-zlib) [ 3105.522516] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 3105.831218] LNet: Added LNI 192.168.204.134@tcp [8/256/0/180] [ 3107.575229] Key type lgssc registered [ 3108.865430] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3118.321068] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3118.355831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3120.152968] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3127.261100] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 3129.330168] LustreError: 91705:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3129.372794] LustreError: 91705:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3134.438405] LustreError: 91706:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3139.560391] LustreError: 91705:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3141.270867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3141.665681] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3147.419076] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 3157.433764] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3162.883953] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3170.804639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3171.246939] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3178.547794] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 3187.174594] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3189.712071] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 3203.055048] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3203.060049] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3203.069371] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3205.090376] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3205.095682] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3210.211131] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3210.217265] Lustre: Skipped 2 previous similar messages [ 3215.333524] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3215.355510] Lustre: Skipped 1 previous similar message [ 3217.377103] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3217.734036] Lustre: server umount lustre-OST0000 complete [ 3225.569120] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3225.571196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3225.588362] Lustre: Skipped 1 previous similar message [ 3231.822590] Lustre: server umount lustre-MDT0000 complete [ 3235.808133] LustreError: 91705:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3235.828354] LustreError: 91705:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3236.049909] LustreError: 91685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789578481 with bad export cookie 16124782311806331541 [ 3236.059028] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3236.068206] LustreError: 91685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3236.589094] Lustre: server umount lustre-MDT0001 complete [ 3249.691867] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [ 3252.552501] Key type lgssc unregistered [ 3252.881523] LNet: 95263:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3252.900809] LNetError: 95263:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3252.923133] LNet: Removed LNI 192.168.204.134@tcp [ 3253.737689] Key type .llcrypt unregistered [ 3253.743261] Key type ._llcrypt unregistered [ 3278.952922] Key type ._llcrypt registered [ 3278.958616] Key type .llcrypt registered [ 3279.124177] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 3280.763362] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3280.799926] alg: No test for adler32 (adler32-zlib) [ 3282.026752] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 3282.248757] LNet: Added LNI 192.168.204.134@tcp [8/256/0/180] [ 3283.922361] Key type lgssc registered [ 3285.210370] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3293.696597] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3293.727570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3309.344376] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3314.463923] LustreError: 96647:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.134@tcp: failed processing log, type 4: rc = -110 [ 3341.152124] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3348.851426] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 3358.479655] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3366.497099] Lustre: server umount lustre-OST0000 complete [ 3380.804236] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 13:10:24 (1789578624) [ 3389.841259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3390.471904] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3395.557101] LustreError: 98289:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3395.574620] LustreError: 98289:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3395.641844] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 3400.674075] LustreError: 98288:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3405.797964] LustreError: 98289:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3406.690389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3407.684416] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3414.614619] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 3424.551807] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3433.026683] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3441.174105] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3441.734380] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3448.711140] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 3458.007614] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3460.225815] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3475.950326] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3475.961204] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3475.962252] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3475.967093] Lustre: Skipped 1 previous similar message [ 3481.071955] LustreError: 100115:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3481.093856] LustreError: 100115:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3481.219300] Lustre: server umount lustre-OST0000 complete [ 3486.180437] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3486.184931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3486.208813] Lustre: Skipped 1 previous similar message [ 3486.227188] Lustre: Skipped 2 previous similar messages [ 3491.297065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3491.319480] Lustre: Skipped 1 previous similar message [ 3491.710956] Lustre: server umount lustre-MDT0000 complete [ 3496.186093] LustreError: 101233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789578742 with bad export cookie 15574620085417011840 [ 3496.190943] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3496.200245] LustreError: 101233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3496.504850] Lustre: server umount lustre-MDT0001 complete [ 3507.870603] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [ 3511.168616] Key type lgssc unregistered [ 3511.521755] LNet: 101983:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3511.529425] LNetError: 101983:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3511.544341] LNet: Removed LNI 192.168.204.134@tcp [ 3512.559099] Key type .llcrypt unregistered [ 3512.561512] Key type ._llcrypt unregistered [ 3558.979186] Key type ._llcrypt registered [ 3558.984145] Key type .llcrypt registered [ 3559.112678] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 3560.089698] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3560.354711] alg: No test for adler32 (adler32-zlib) [ 3561.510709] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 3561.829493] LNet: Added LNI 192.168.204.134@tcp [8/256/0/180] [ 3563.575626] Key type lgssc registered [ 3565.038425] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3574.628822] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3574.646627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3576.210895] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3581.852963] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 3586.542855] LustreError: 104169:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3586.566812] LustreError: 104169:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3587.559314] LustreError: 104168:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3591.899537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3592.362021] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3597.175044] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 3605.580501] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3612.465894] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3620.666107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3621.010366] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3623.094373] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3627.694834] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 3636.330749] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3647.996872] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3648.014715] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3648.039297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3652.043111] Lustre: server umount lustre-OST0000 complete [ 3658.210275] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3658.215945] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3658.233244] Lustre: Skipped 2 previous similar messages [ 3658.244562] Lustre: Skipped 1 previous similar message [ 3663.330479] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3663.340226] Lustre: Skipped 2 previous similar messages [ 3663.700949] Lustre: server umount lustre-MDT0000 complete [ 3667.427186] LustreError: 105389:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3667.459795] LustreError: 105389:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3668.145450] LustreError: 106953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789578914 with bad export cookie 9781956915064756988 [ 3668.146927] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3668.175424] LustreError: 106953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3668.698077] Lustre: server umount lustre-MDT0001 complete [ 3684.676810] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [ 3687.700637] Key type lgssc unregistered [ 3688.147946] LNet: 107704:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3688.169852] LNetError: 107704:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3688.203581] LNet: Removed LNI 192.168.204.134@tcp [ 3689.257235] Key type .llcrypt unregistered [ 3689.258717] Key type ._llcrypt unregistered [ 3711.807505] Key type ._llcrypt registered [ 3711.809629] Key type .llcrypt registered [ 3711.912659] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 3713.259727] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3713.329879] alg: No test for adler32 (adler32-zlib) [ 3714.495876] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 3714.735930] LNet: Added LNI 192.168.204.134@tcp [8/256/0/180] [ 3716.455332] Key type lgssc registered [ 3717.488360] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3726.641890] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3726.679545] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3742.240616] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3747.359482] LustreError: 108942:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.134@tcp: failed processing log, type 4: rc = -110 [ 3774.047430] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3780.656557] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 3789.437232] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3797.193909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3797.587897] LustreError: MGC192.168.204.134@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3797.602614] LustreError: 109968:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3797.610848] LustreError: 109968:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 3797.618968] LustreError: 109968:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 3797.710189] Lustre: server umount lustre-MDT0000 complete [ 3797.718534] LustreError: 109968:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3817.951632] Lustre: 108278:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789579047/real 1789579047] req@000000008ef79604 x1876509547564416/t0(0) o400->MGC192.168.204.134@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789579063 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3818.002709] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3818.055685] LustreError: 108954:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3838.582782] Lustre: server umount lustre-OST0000 complete [ 3850.103154] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_hostid [ 3862.371791] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 3902.914089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3903.257971] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3903.313581] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3903.446757] Lustre: lustre-MDT0000: new disk, initializing [ 3903.597100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3903.621387] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3908.868518] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 3922.716237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3922.801971] Lustre: 112764:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3922.897106] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3922.901335] Lustre: Skipped 1 previous similar message [ 3922.957895] Lustre: lustre-MDT0001: new disk, initializing [ 3923.125927] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3923.188735] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3923.204058] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3930.248529] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 3945.958909] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3956.048346] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3964.392104] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3964.682545] Lustre: lustre-OST0000: new disk, initializing [ 3964.686675] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3964.696315] Lustre: 114736:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3964.799429] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3971.150584] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3971.157289] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3971.321156] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3972.344356] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 3984.361480] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3989.822455] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3990.129363] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3995.270113] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3995.568918] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4002.286796] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4002.297129] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4002.303814] LustreError: Skipped 1 previous similar message [ 4002.309386] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4002.324960] Lustre: Skipped 1 previous similar message [ 4002.367516] Lustre: Skipped 1 previous similar message [ 4004.435527] Lustre: server umount lustre-OST0000 complete [ 4010.467572] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4010.472830] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4010.494040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4012.514374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4012.514593] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4017.638472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4017.653142] Lustre: Skipped 2 previous similar messages [ 4022.755850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4022.767857] Lustre: Skipped 1 previous similar message [ 4023.264080] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4023.527484] Lustre: server umount lustre-MDT0000 complete [ 4028.911414] LustreError: 112756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789579274 with bad export cookie 11825961236106669242 [ 4028.933976] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4028.935950] LustreError: 112756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4029.580618] Lustre: server umount lustre-MDT0001 complete [ 4040.638321] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 13:21:25 (1789579285) [ 4045.370069] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 4066.876973] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_hostid [ 4080.024268] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 4123.270814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4123.458250] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4123.494121] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4123.642927] Lustre: lustre-MDT0000: new disk, initializing [ 4123.802424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4123.829078] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4128.741625] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 4141.508130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4141.711513] Lustre: 119644:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4141.846832] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4141.851068] Lustre: Skipped 1 previous similar message [ 4141.987623] Lustre: lustre-MDT0001: new disk, initializing [ 4142.170890] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4142.207524] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4142.220106] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4147.077957] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 4157.208578] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4164.285842] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4172.590684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4173.231303] Lustre: lustre-OST0000: new disk, initializing [ 4173.234324] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4173.238818] Lustre: 121616:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4173.352558] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4180.128776] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 4182.102824] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4182.125066] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4182.281864] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4191.045980] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4197.305899] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4197.499666] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4202.219285] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4202.483444] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4208.102684] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4208.103792] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4208.147929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4208.177812] Lustre: Skipped 1 previous similar message [ 4219.359199] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4219.603469] Lustre: server umount lustre-OST0000 complete [ 4228.577696] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4228.606127] Lustre: Skipped 1 previous similar message [ 4228.613044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4228.618517] Lustre: Skipped 5 previous similar messages [ 4230.686212] Lustre: server umount lustre-MDT0000 complete [ 4233.697162] LustreError: 119657:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4233.738754] LustreError: 119657:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 4234.851437] LustreError: 120439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789579480 with bad export cookie 11825961236106670124 [ 4234.861601] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4234.863558] LustreError: 120439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4235.231286] Lustre: server umount lustre-MDT0001 complete [ 4245.890445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4246.604494] LustreError: 123835:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4246.719902] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4252.134115] LustreError: 123836:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4252.836386] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 4256.233824] LustreError: 123835:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4261.346758] LustreError: 123836:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4263.599541] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4269.889930] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 4278.510164] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4285.524785] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4293.236256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4302.056857] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 4311.203679] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4323.816984] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 4329.957471] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4329.972968] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4329.995055] Lustre: Skipped 1 previous similar message [ 4330.009606] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4334.567068] Lustre: server umount lustre-OST0000 complete [ 4340.712858] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4340.731601] LustreError: Skipped 1 previous similar message [ 4345.277822] Lustre: server umount lustre-MDT0000 complete [ 4347.366815] LustreError: 123831:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4347.392734] LustreError: 123831:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4349.731698] LustreError: 123816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789579595 with bad export cookie 11825961236106671006 [ 4349.735650] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4349.749086] LustreError: 123816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4350.202705] Lustre: server umount lustre-MDT0001 complete [ 4365.010672] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [ 4368.529801] Key type lgssc unregistered [ 4368.842794] LNet: 127465:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4368.851488] LNetError: 127465:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4368.869444] LNet: Removed LNI 192.168.204.134@tcp [ 4369.676860] Key type .llcrypt unregistered [ 4369.678694] Key type ._llcrypt unregistered [ 4389.466268] Key type ._llcrypt registered [ 4389.469328] Key type .llcrypt registered [ 4389.640500] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_hostid [ 4406.308093] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 4407.511899] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4407.587921] alg: No test for adler32 (adler32-zlib) [ 4408.735633] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 4409.012639] LNet: Added LNI 192.168.204.134@tcp [8/256/0/180] [ 4410.695370] Key type lgssc registered [ 4411.938449] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4453.450435] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4453.483532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4454.955661] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4455.081267] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4455.237327] Lustre: lustre-MDT0000: new disk, initializing [ 4455.404715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4455.445232] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4464.748607] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 4478.436741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4478.509667] Lustre: 130616:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4478.597670] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4478.600981] Lustre: Skipped 1 previous similar message [ 4478.666196] Lustre: lustre-MDT0001: new disk, initializing [ 4478.772625] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4478.809711] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4478.831506] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4482.736063] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 4492.860717] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4498.999646] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4506.668314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4506.958898] Lustre: lustre-OST0000: new disk, initializing [ 4506.964215] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4506.971341] Lustre: 132588:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4507.046404] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4513.936532] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4513.962506] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4514.088268] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4514.366724] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 4524.307660] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4529.098697] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4529.283366] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4534.055154] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4534.319332] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4539.881332] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4539.882767] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4539.894138] LustreError: Skipped 1 previous similar message [ 4539.895375] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4539.941470] Lustre: Skipped 1 previous similar message [ 4542.606724] Lustre: server umount lustre-OST0000 complete [ 4550.114503] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4550.139053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4550.147642] Lustre: Skipped 2 previous similar messages [ 4553.023288] Lustre: server umount lustre-MDT0000 complete [ 4557.688909] LustreError: 130608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789579803 with bad export cookie 13727882898686543289 [ 4557.693741] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4557.720465] LustreError: 130608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4558.386428] Lustre: server umount lustre-MDT0001 complete [ 4569.415896] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 13:30:14 (1789579814) [ 4578.417424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4579.198668] LustreError: 135008:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4579.212217] LustreError: 135008:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4579.325774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4584.418337] LustreError: 135009:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4584.998111] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 4589.539726] LustreError: 135008:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4594.665367] LustreError: 135009:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4596.762816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4597.160983] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4601.584776] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 4609.834699] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4618.238091] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4621.671520] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4621.678885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4621.691547] Lustre: Skipped 2 previous similar messages [ 4626.912291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4626.921281] Lustre: Skipped 2 previous similar messages [ 4626.988457] Lustre: server umount lustre-MDT0000 complete [ 4631.090822] LustreError: 134990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789579876 with bad export cookie 13727882898686544171 [ 4631.104534] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4631.108959] LustreError: 134990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4631.659147] Lustre: server umount lustre-MDT0001 complete [ 4642.718315] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 13:31:27 (1789579887) [ 4652.888255] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4668.640594] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4673.759548] LustreError: 137424:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.134@tcp: failed processing log, type 4: rc = -110 [ 4699.488918] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4705.522296] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 4715.123874] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4718.360616] Lustre: server umount lustre-OST0000 complete [ 4730.450993] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 13:32:55 (1789579975) [ 4739.388145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4739.888723] LustreError: 138871:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4739.968693] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4746.020187] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 4749.284421] LustreError: 138872:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4749.309749] LustreError: 138872:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4756.197516] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4761.071187] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 4771.825871] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4779.330024] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4787.435278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4787.943759] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4787.956668] Lustre: Skipped 1 previous similar message [ 4794.393869] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 4802.416845] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4823.011931] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4823.021966] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4823.045545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4826.707913] Lustre: server umount lustre-MDT0000 complete [ 4830.179232] LustreError: 139877:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4830.205499] LustreError: 139877:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4830.596364] LustreError: 138853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789580076 with bad export cookie 13727882898686544836 [ 4830.603619] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4830.621556] LustreError: 138853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4830.988529] Lustre: server umount lustre-MDT0001 complete [ 4845.872898] Lustre: server umount lustre-OST0000 complete [ 4854.684470] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 13:34:59 (1789580099) [ 4862.868188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4863.303300] LustreError: 142552:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4863.426535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4868.652868] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 4879.433217] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4886.472486] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 4897.929038] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4905.561035] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4913.321588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4921.563557] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 4929.831441] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4934.206346] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4935.322659] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4935.516546] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4939.838166] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4940.033796] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4944.359045] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4944.363316] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4944.384389] Lustre: Skipped 2 previous similar messages [ 4944.400615] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4944.405761] Lustre: Skipped 4 previous similar messages [ 4956.127275] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4956.296930] Lustre: server umount lustre-OST0000 complete [ 4961.760864] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4961.776720] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4961.788068] Lustre: Skipped 1 previous similar message [ 4961.802803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4961.815043] Lustre: Skipped 4 previous similar messages [ 4967.810336] Lustre: server umount lustre-MDT0000 complete [ 4969.965896] LustreError: 142553:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4970.017150] LustreError: 142553:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 4974.798531] LustreError: 142533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789580220 with bad export cookie 13727882898686545907 [ 4974.800723] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4974.828854] LustreError: 142533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4975.424786] Lustre: server umount lustre-MDT0001 complete [ 4988.929150] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 13:37:13 (1789580233) [ 4997.158290] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5012.767721] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5017.887532] LustreError: 146713:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.134@tcp: failed processing log, type 4: rc = -110 [ 5043.551605] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5043.559724] Lustre: Skipped 2 previous similar messages [ 5051.016329] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 5060.611383] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5069.063190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5069.867953] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 5075.287988] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 5084.592528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5088.769573] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 5096.117779] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5102.737628] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5107.955602] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5108.153768] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5113.407498] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5113.636426] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5120.996132] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5120.999312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5121.001744] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5121.016169] Lustre: Skipped 3 previous similar messages [ 5121.039482] Lustre: Skipped 4 previous similar messages [ 5122.422655] Lustre: server umount lustre-MDT0000 complete [ 5126.126436] LustreError: 149002:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5126.152326] LustreError: 149002:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 5126.784848] LustreError: 146720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789580372 with bad export cookie 13727882898686546929 [ 5126.792639] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5126.803090] LustreError: 146720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5127.182737] Lustre: server umount lustre-MDT0001 complete [ 5141.825891] Lustre: server umount lustre-OST0000 complete [ 5152.674665] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 13:39:57 (1789580397) [ 5160.838258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5176.416896] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5181.535473] LustreError: 150911:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.134@tcp: failed processing log, type 4: rc = -110 [ 5213.906810] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 5222.436116] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5229.028228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5229.774239] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 5234.615504] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 5244.547164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5251.041535] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 5264.751076] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5273.230672] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5283.064113] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5283.367368] Lustre: lustre-OST0001: new disk, initializing [ 5283.372674] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5283.387608] Lustre: 153788:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5290.912443] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 5291.083452] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5291.093689] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5291.290368] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5300.781359] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5305.704989] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5305.960148] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5311.336981] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5311.606971] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5316.581587] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5316.600579] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5316.625870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5328.351294] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5328.477587] Lustre: server umount lustre-OST0000 complete [ 5338.630355] Lustre: server umount lustre-OST0001 complete [ 5348.911216] Lustre: server umount lustre-MDT0000 complete [ 5353.187335] LustreError: 150917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789580599 with bad export cookie 13727882898686547881 [ 5353.189117] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5353.214123] LustreError: 150917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5353.805816] Lustre: server umount lustre-MDT0001 complete [ 5383.644397] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 13:43:48 (1789580628) [ 5385.625172] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 5387.948414] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 5389.875433] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 13:43:54 (1789580634) [ 5397.296982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5397.580777] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5397.643865] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5397.817231] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5397.829548] Lustre: Skipped 6 previous similar messages [ 5402.721973] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 5412.717534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5412.829842] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5412.866963] Lustre: 157348:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5417.587392] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 5426.598826] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5434.262274] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5443.591896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5443.810032] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5452.141550] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 5454.026668] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 5461.337032] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5467.834392] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5468.127570] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5474.357326] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5474.651350] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5479.392727] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5479.401231] LustreError: Skipped 3 previous similar messages [ 5479.406262] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5479.422937] Lustre: Skipped 5 previous similar messages [ 5479.435104] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5479.443595] Lustre: Skipped 14 previous similar messages [ 5483.239367] Lustre: server umount lustre-OST0000 complete [ 5507.780459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5516.948949] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 5519.541088] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5519.574338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5519.592825] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5519.600411] Lustre: Skipped 1 previous similar message [ 5526.749715] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5533.098863] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5533.431416] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5539.136701] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5539.465697] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5546.505813] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff95b00798f800.ost_server_uuid 50 [ 5548.548843] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff95b00798f800.ost_server_uuid in FULL state after 0 sec [ 5548.807432] LustreError: 157357:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.204.34@tcp) returned error from blocking AST (req@0000000077460cb5 x1876510275555328 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000deb11f74/0xbe833cc34e5f2103 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.34@tcp remote: 0x6d69f0a47d0ec211 expref: 5 pid: 157356 timeout: 5648 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5548.859872] LustreError: lustre-MDT0000: A client on nid 192.168.204.34@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5548.870570] LustreError: 157345:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.204.34@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000deb11f74/0xbe833cc34e5f2103 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.34@tcp remote: 0x6d69f0a47d0ec211 expref: 6 pid: 157356 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5555.182893] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5555.192073] LustreError: Skipped 1 previous similar message [ 5560.817668] LustreError: 160979:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5560.846871] LustreError: 160979:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 5561.115129] Lustre: server umount lustre-OST0000 complete [ 5580.255118] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5580.424549] Lustre: server umount lustre-MDT0000 complete [ 5586.748620] LustreError: 157340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789580832 with bad export cookie 13727882898686549064 [ 5586.785854] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5586.795440] LustreError: 157340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5601.247773] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5601.606994] Lustre: server umount lustre-MDT0001 complete [ 5614.889694] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [ 5618.101843] Key type lgssc unregistered [ 5618.479092] LNet: 163467:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5618.495649] LNetError: 163467:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5618.522847] LNet: Removed LNI 192.168.204.134@tcp [ 5619.608089] Key type .llcrypt unregistered [ 5619.610982] Key type ._llcrypt unregistered [ 5647.195919] Key type ._llcrypt registered [ 5647.198860] Key type .llcrypt registered [ 5647.308982] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 5648.665283] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5648.782978] alg: No test for adler32 (adler32-zlib) [ 5649.786552] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 5650.130512] LNet: Added LNI 192.168.204.134@tcp [8/256/0/180] [ 5651.831693] Key type lgssc registered [ 5653.144429] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5662.123662] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5662.172783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5663.576443] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5668.512301] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 5668.839047] LustreError: 164877:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5668.853214] LustreError: 164877:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5673.951612] LustreError: 164878:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5678.330197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5678.863222] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5684.414477] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 5697.032141] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5704.998139] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5714.827234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5715.312852] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5723.707154] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 5733.768943] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5734.126538] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5742.162735] Lustre: Failing over lustre-MDT0000 [ 5742.424958] LustreError: 167458:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x736d61726170:0x3:0x0].0x0 (0000000066c684a8) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 5742.767508] Lustre: server umount lustre-MDT0000 complete [ 5744.104806] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5744.116613] LustreError: 164878:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5744.124416] Lustre: Skipped 1 previous similar message [ 5744.147644] LustreError: 164878:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5749.224286] LustreError: 165605:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5749.254243] LustreError: 165605:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5754.336951] LustreError: 164877:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5754.357875] LustreError: 164877:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5755.059228] LustreError: 164858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.34@tcp arrived at 1789581000 with bad export cookie 17564576646802436357 [ 5755.094889] LustreError: 164858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5759.398169] Lustre: 164187:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789580989/real 1789580989] req@0000000043faadca x1876511577633280/t0(0) o400->MGC192.168.204.134@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789581005 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5759.441042] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5763.679331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5764.583264] LustreError: 164872:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5764.601595] LustreError: 164872:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 5769.714480] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xf3c1e9662ec7f167 [ 5769.729305] Lustre: MGC192.168.204.134@tcp: Connection restored to 0@lo (at 0@lo) [ 5770.107683] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5770.219513] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5774.883366] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 5775.358936] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5775.371246] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5777.626172] Lustre: lustre-MDT0000: Denying connection for new client 5877dbaf-9f99-4f58-9a9b-b5997d7a80ca (at 192.168.204.34@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:07 [ 5782.825878] Lustre: lustre-MDT0000: Denying connection for new client 5877dbaf-9f99-4f58-9a9b-b5997d7a80ca (at 192.168.204.34@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:02 [ 5790.180127] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5790.199537] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5790.223884] Lustre: Skipped 1 previous similar message [ 5790.236034] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5792.424413] Lustre: server umount lustre-OST0000 complete [ 5797.576261] LustreError: 168613:0:(ldlm_lib.c:3038:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5797.593752] Lustre: 167963:0:(ldlm_lib.c:2438:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5797.610921] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5797.637932] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5797.658524] LustreError: 167963:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000000dddd08e x1876511577653888/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5797.702432] LustreError: 167963:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5797.714648] LustreError: 167963:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5797.752612] Lustre: lustre-MDT0000: Recovery over after 0:22, of 2 clients 0 recovered and 2 were evicted. [ 5798.072715] Lustre: server umount lustre-MDT0000 complete [ 5800.928293] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5800.941849] Lustre: Skipped 1 previous similar message [ 5800.952677] LustreError: 164874:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5800.992254] LustreError: 164874:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 5803.358001] LustreError: 167574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789581049 with bad export cookie 17564576646802436455 [ 5803.358510] LustreError: 164185:0:(import.c:752:ptlrpc_connect_import_locked()) can't connect to a closed import [ 5803.363619] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5803.377097] LustreError: 167574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5809.925889] Lustre: server umount lustre-MDT0001 complete [ 5826.612954] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [ 5830.675336] Key type lgssc unregistered [ 5831.013030] LNet: 169366:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5831.027067] LNetError: 169366:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5831.045615] LNet: Removed LNI 192.168.204.134@tcp [ 5832.276631] Key type .llcrypt unregistered [ 5832.281574] Key type ._llcrypt unregistered [ 5862.430126] Key type ._llcrypt registered [ 5862.431874] Key type .llcrypt registered [ 5862.512870] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 5863.341589] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5863.374110] alg: No test for adler32 (adler32-zlib) [ 5864.537829] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 5865.095448] LNet: Added LNI 192.168.204.134@tcp [8/256/0/180] [ 5866.871259] Key type lgssc registered [ 5868.846711] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5881.180634] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5881.273411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5883.070212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5890.089677] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 5892.583800] LustreError: 170779:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5892.604112] LustreError: 170779:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5897.704243] LustreError: 170780:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5900.996279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5901.392498] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5906.329368] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 5915.143198] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5921.935055] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5929.663516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5929.933571] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5936.053487] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 5943.731538] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5952.443211] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5961.701059] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5961.709696] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5961.731118] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5962.725554] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5962.727217] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5962.728389] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5962.733807] Lustre: Skipped 1 previous similar message [ 5967.313216] Lustre: server umount lustre-OST0000 complete [ 5972.962178] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5972.969659] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5972.983493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5977.814559] Lustre: server umount lustre-MDT0000 complete [ 5978.084462] LustreError: 170779:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5978.118102] LustreError: 170779:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5982.679441] LustreError: 170759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789581228 with bad export cookie 7339916247684351855 [ 5982.698846] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5983.073913] Lustre: server umount lustre-MDT0001 complete [ 5998.389036] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [ 6000.956502] Key type lgssc unregistered [ 6001.221767] LNet: 174335:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6001.229960] LNetError: 174335:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6001.247300] LNet: Removed LNI 192.168.204.134@tcp [ 6002.284341] Key type .llcrypt unregistered [ 6002.287550] Key type ._llcrypt unregistered [ 6042.835858] Key type ._llcrypt registered [ 6042.839642] Key type .llcrypt registered [ 6043.001748] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 6044.395469] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6044.571728] alg: No test for adler32 (adler32-zlib) [ 6045.750419] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 6046.099623] LNet: Added LNI 192.168.204.134@tcp [8/256/0/180] [ 6047.863322] Key type lgssc registered [ 6049.101335] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6057.215569] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6057.236242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6058.760636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6063.798852] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 6068.196622] LustreError: 176042:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6068.227921] LustreError: 176042:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6073.316542] LustreError: 176041:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6073.473889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6073.832175] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6078.284753] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 6086.702692] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6094.929298] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6103.787363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6104.377860] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6112.975295] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 6126.585351] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6129.028553] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 6144.486633] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6144.495560] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6144.514786] LustreError: Skipped 1 previous similar message [ 6144.518734] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6144.536395] Lustre: Skipped 1 previous similar message [ 6144.567981] Lustre: Skipped 1 previous similar message [ 6145.939872] Lustre: server umount lustre-OST0000 complete [ 6154.720500] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6154.726593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6154.742516] Lustre: Skipped 1 previous similar message [ 6154.780842] Lustre: Skipped 1 previous similar message [ 6158.087988] Lustre: server umount lustre-MDT0000 complete [ 6159.841453] LustreError: 176041:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6159.875398] LustreError: 176041:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6162.606064] LustreError: 176023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789581408 with bad export cookie 12517433372482191681 [ 6162.609385] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6162.613082] LustreError: 176023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6162.869486] Lustre: server umount lustre-MDT0001 complete [ 6180.254065] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [ 6184.026736] Key type lgssc unregistered [ 6184.407767] LNet: 179619:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6184.419495] LNetError: 179619:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6184.429258] LNet: Removed LNI 192.168.204.134@tcp [ 6185.591558] Key type .llcrypt unregistered [ 6185.593341] Key type ._llcrypt unregistered [ 6215.382397] Key type ._llcrypt registered [ 6215.385369] Key type .llcrypt registered [ 6215.527239] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 6216.536834] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6216.774852] alg: No test for adler32 (adler32-zlib) [ 6217.960673] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 6218.244968] LNet: Added LNI 192.168.204.134@tcp [8/256/0/180] [ 6219.903259] Key type lgssc registered [ 6221.353350] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6233.852567] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6233.892835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6235.435155] Lustre: *** cfs_fail_loc=135, val=0*** [ 6235.608596] LustreError: 181141:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 6235.628415] LustreError: 181141:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.134@tcp: cfg command failed: rc = -2 [ 6235.653388] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 6235.653388] [ 6235.663695] LustreError: MGC192.168.204.134@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 6235.694761] LustreError: 181103:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 6235.711377] LustreError: 181103:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 6235.720660] LustreError: 181103:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 6235.730993] LustreError: 181103:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 6235.828030] Lustre: server umount lustre-MDT0000 complete [ 6235.830543] LustreError: 181103:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 6244.926454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6245.588248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6250.831393] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 6256.097294] LustreError: 181514:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6256.116413] LustreError: 181514:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6260.785261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6261.117828] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6266.918398] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 6275.986375] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6283.997791] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6290.912946] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6290.915180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6290.927892] Lustre: Skipped 1 previous similar message [ 6290.939793] Lustre: Skipped 1 previous similar message [ 6293.385464] Lustre: server umount lustre-MDT0000 complete [ 6296.037890] LustreError: 181514:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6296.057205] LustreError: 181514:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6299.101058] LustreError: 181494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789581544 with bad export cookie 8831749834927357490 [ 6299.109496] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6299.133406] LustreError: 181494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6299.892224] Lustre: server umount lustre-MDT0001 complete [ 6318.249885] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [ 6321.328410] Key type lgssc unregistered [ 6321.653994] LNet: 183855:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6321.661112] LNetError: 183855:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6321.672765] LNet: Removed LNI 192.168.204.134@tcp [ 6322.369239] Key type .llcrypt unregistered [ 6322.371556] Key type ._llcrypt unregistered [ 6349.882959] Key type ._llcrypt registered [ 6349.891816] Key type .llcrypt registered [ 6350.145672] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [ 6350.738268] Key type .llcrypt unregistered [ 6350.740138] Key type ._llcrypt unregistered [ 6373.576689] Key type ._llcrypt registered [ 6373.582040] Key type .llcrypt registered [ 6373.765975] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 6375.976394] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6376.010467] alg: No test for adler32 (adler32-zlib) [ 6377.303225] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 6377.642642] LNet: Added LNI 192.168.204.134@tcp [8/256/0/180] [ 6379.353282] Key type lgssc registered [ 6380.403855] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6389.021792] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6389.054802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6404.639495] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6409.759400] LustreError: 185985:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.134@tcp: failed processing log, type 4: rc = -110 [ 6436.447280] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6443.310755] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 6452.305668] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6461.001857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6461.623488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6461.722969] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 6467.059092] LustreError: 186010:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6467.102959] LustreError: 186010:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6467.877780] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 6471.203616] LustreError: 187032:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6471.243186] LustreError: 187032:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6476.258783] LustreError: 187033:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6476.298036] LustreError: 187033:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6480.082946] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6480.594516] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6486.727790] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 6498.208483] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6504.676790] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6507.762605] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6518.240792] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6518.248394] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6518.261431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6521.827431] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6521.828415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6521.839265] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6523.477855] Lustre: server umount lustre-OST0000 complete [ 6532.065311] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6532.070197] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6532.101307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6532.117199] Lustre: Skipped 2 previous similar messages [ 6533.800595] Lustre: server umount lustre-MDT0000 complete [ 6537.185476] LustreError: 187037:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6537.218491] LustreError: 187037:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6538.589447] LustreError: 185992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789581784 with bad export cookie 7477458712170729973 [ 6538.594934] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6538.605392] LustreError: 185992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6539.152274] Lustre: server umount lustre-MDT0001 complete [ 6556.949198] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [ 6560.635599] Key type lgssc unregistered [ 6560.975974] LNet: 189926:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6560.980425] LNetError: 189926:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6561.015623] LNet: Removed LNI 192.168.204.134@tcp [ 6562.457908] Key type .llcrypt unregistered [ 6562.462924] Key type ._llcrypt unregistered [ 6591.729828] Key type ._llcrypt registered [ 6591.735465] Key type .llcrypt registered [ 6591.867677] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 6593.159733] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6593.192561] alg: No test for adler32 (adler32-zlib) [ 6594.493803] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 6595.091456] LNet: Added LNI 192.168.204.134@tcp [8/256/0/180] [ 6597.023217] Key type lgssc registered [ 6598.308244] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6608.122481] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6608.140414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6609.740518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6615.062852] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 6620.136143] LustreError: 191338:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6620.162938] LustreError: 191338:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6624.666078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6625.212091] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6629.729564] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 6638.854297] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6645.827647] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6654.949801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6655.373583] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6656.435121] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 6663.798061] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 6673.541224] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6681.115311] Lustre: Failing over lustre-MDT0000 [ 6681.434777] LustreError: 191334:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6681.569882] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6681.575415] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6681.644359] Lustre: server umount lustre-MDT0000 complete [ 6685.157280] LustreError: 191338:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6685.160051] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6685.186162] LustreError: 191338:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6685.203172] Lustre: Skipped 1 previous similar message [ 6690.279599] LustreError: 191333:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6690.322060] LustreError: 191333:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 6695.394949] LustreError: 192065:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6695.434579] LustreError: 192065:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 6701.445663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6701.535110] Lustre: 190647:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789581931/real 1789581931] req@0000000042400af6 x1876512567489408/t0(0) o400->MGC192.168.204.134@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789581947 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6701.572919] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6706.668810] LustreError: 194103:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6706.686954] LustreError: 194103:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [ 6711.809582] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x7dfd6e66e7e8280f [ 6711.824755] Lustre: MGC192.168.204.134@tcp: Connection restored to 0@lo (at 0@lo) [ 6712.271946] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6712.400716] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6714.187920] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6717.461122] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6717.486639] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 6717.545932] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 6718.376574] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 6724.108871] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6732.192546] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6746.085700] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 6746.092159] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6746.105304] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6747.596333] Lustre: server umount lustre-OST0000 complete [ 6753.251035] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6753.254575] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6753.269347] LustreError: Skipped 1 previous similar message [ 6753.291705] Lustre: Skipped 1 previous similar message [ 6753.295223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6753.295230] Lustre: Skipped 1 previous similar message [ 6758.149406] Lustre: server umount lustre-MDT0000 complete [ 6758.375549] LustreError: 191337:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6758.405469] LustreError: 191337:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 6762.418437] LustreError: 195173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789582008 with bad export cookie 9078533812105259023 [ 6762.428666] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6762.434833] LustreError: 195173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6763.001722] Lustre: server umount lustre-MDT0001 complete [ 6775.786400] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [ 6779.962096] Key type lgssc unregistered [ 6780.472813] LNet: 196257:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6780.502871] LNetError: 196257:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6780.520883] LNet: Removed LNI 192.168.204.134@tcp [ 6781.409979] Key type .llcrypt unregistered [ 6781.413638] Key type ._llcrypt unregistered [ 6812.856365] Key type ._llcrypt registered [ 6812.859185] Key type .llcrypt registered [ 6812.966604] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 6814.206240] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6814.268756] alg: No test for adler32 (adler32-zlib) [ 6815.513686] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 6815.734372] LNet: Added LNI 192.168.204.134@tcp [8/256/0/180] [ 6817.439241] Key type lgssc registered [ 6818.680850] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6826.895465] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6826.912920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6828.430163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6833.171392] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 6838.756215] LustreError: 197667:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6838.772386] LustreError: 197667:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6841.670450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6841.927762] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6846.748596] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 6854.984046] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6861.771063] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6868.427309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6868.847518] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6874.986261] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6875.780613] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 6884.022490] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6887.741849] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6892.371148] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6912.636100] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6921.195683] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6921.214950] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6921.248456] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6922.211299] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6922.231997] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6922.245391] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6922.253074] Lustre: Skipped 1 previous similar message [ 6924.714934] Lustre: server umount lustre-OST0000 complete [ 6932.449301] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6932.452602] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6932.469120] Lustre: Skipped 1 previous similar message [ 6932.480840] Lustre: Skipped 1 previous similar message [ 6936.014222] Lustre: server umount lustre-MDT0000 complete [ 6937.571703] LustreError: 198394:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6940.310726] LustreError: 200200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789582186 with bad export cookie 16429950477169906313 [ 6940.314419] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6940.316361] LustreError: 200200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6940.877670] Lustre: server umount lustre-MDT0001 complete [ 6953.192622] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [ 6956.836929] Key type lgssc unregistered [ 6957.281121] LNet: 201532:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6957.288631] LNetError: 201532:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6957.307829] LNet: Removed LNI 192.168.204.134@tcp [ 6958.818191] Key type .llcrypt unregistered [ 6958.820349] Key type ._llcrypt unregistered [ 6981.272988] Key type ._llcrypt registered [ 6981.277522] Key type .llcrypt registered [ 6981.392558] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 6982.909776] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6982.979921] alg: No test for adler32 (adler32-zlib) [ 6984.157624] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 6984.420816] LNet: Added LNI 192.168.204.134@tcp [8/256/0/180] [ 6986.143248] Key type lgssc registered [ 6987.377389] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6996.160715] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6996.177605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6997.687250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7002.644276] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 7007.203546] LustreError: 202796:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7007.218830] LustreError: 202796:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7012.271640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7012.323086] LustreError: 202797:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7012.755570] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7017.397940] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 7025.542773] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7032.288356] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7039.134372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7039.680719] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7042.752639] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 7045.789578] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 7054.422124] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7061.623158] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 7070.791181] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 7081.759371] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 7091.914810] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 7109.605685] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7109.609966] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7109.647819] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7109.677300] LustreError: Skipped 1 previous similar message [ 7111.684883] Lustre: server umount lustre-OST0000 complete [ 7119.852489] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7119.853517] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7119.889696] Lustre: Skipped 1 previous similar message [ 7119.901726] Lustre: Skipped 2 previous similar messages [ 7122.286455] Lustre: server umount lustre-MDT0000 complete [ 7124.962582] LustreError: 202797:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7124.999288] LustreError: 202797:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7126.254093] LustreError: 202777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789582372 with bad export cookie 6125027100835183088 [ 7126.257338] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7126.267572] LustreError: 202777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7126.839236] Lustre: server umount lustre-MDT0001 complete [ 7140.610792] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [ 7144.701217] Key type lgssc unregistered [ 7144.978458] LNet: 207867:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7144.984045] LNetError: 207867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7145.031557] LNet: Removed LNI 192.168.204.134@tcp [ 7146.154042] Key type .llcrypt unregistered [ 7146.161466] Key type ._llcrypt unregistered [ 7176.230841] Key type ._llcrypt registered [ 7176.233691] Key type .llcrypt registered [ 7176.387115] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 7177.695179] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7177.914405] alg: No test for adler32 (adler32-zlib) [ 7179.035833] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 7179.306418] LNet: Added LNI 192.168.204.134@tcp [8/256/0/180] [ 7181.103225] Key type lgssc registered [ 7182.264388] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7191.356685] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7191.375213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7192.884368] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7197.431266] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 7202.276304] LustreError: 209278:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7202.294192] LustreError: 209278:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7207.395728] LustreError: 209279:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7207.620143] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7207.946445] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7212.448184] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 7222.283927] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7229.098432] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7236.892747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7237.288576] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7238.332548] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 7243.943110] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 7252.811329] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7273.448716] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7273.455829] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7273.460800] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7273.476491] Lustre: Skipped 1 previous similar message [ 7277.458499] Lustre: server umount lustre-OST0000 complete [ 7283.686242] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7283.709604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7283.717637] Lustre: Skipped 2 previous similar messages [ 7288.072166] Lustre: server umount lustre-MDT0000 complete [ 7288.801891] LustreError: 210880:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7288.817435] LustreError: 210880:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7292.418464] LustreError: 209258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789582538 with bad export cookie 14244526762786381615 [ 7292.429219] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7292.443413] LustreError: 209258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7292.898954] Lustre: server umount lustre-MDT0001 complete [ 7309.538954] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [ 7314.375796] Key type lgssc unregistered [ 7314.853749] LNet: 212964:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7314.860196] LNetError: 212964:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7314.887739] LNet: Removed LNI 192.168.204.134@tcp [ 7315.998661] Key type .llcrypt unregistered [ 7316.006478] Key type ._llcrypt unregistered [ 7339.404260] Key type ._llcrypt registered [ 7339.411765] Key type .llcrypt registered [ 7339.542196] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 7340.864837] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7340.909750] alg: No test for adler32 (adler32-zlib) [ 7342.027867] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 7342.265180] LNet: Added LNI 192.168.204.134@tcp [8/256/0/180] [ 7343.967291] Key type lgssc registered [ 7345.313520] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7352.946923] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7352.974624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7354.452643] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7359.150703] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 7364.591367] LustreError: 214230:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7364.610456] LustreError: 214230:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7369.624389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7369.706264] LustreError: 214229:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7370.066542] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7375.008394] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 7383.872897] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7390.910580] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7398.768870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7399.150582] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7400.219982] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 7406.338973] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 7414.764984] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7430.115682] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7430.148697] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7434.315169] Lustre: server umount lustre-OST0000 complete [ 7440.354323] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7440.385520] Lustre: Skipped 1 previous similar message [ 7440.391432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7440.395032] Lustre: Skipped 2 previous similar messages [ 7444.805327] Lustre: server umount lustre-MDT0000 complete [ 7445.477392] LustreError: 215415:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7445.492963] LustreError: 215415:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7448.630973] LustreError: 214209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789582694 with bad export cookie 5031096203844809422 [ 7448.635554] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7448.640579] LustreError: 214209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7449.045094] Lustre: server umount lustre-MDT0001 complete [ 7466.780160] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [ 7470.329756] Key type lgssc unregistered [ 7470.731607] LNet: 217927:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7470.744566] LNetError: 217927:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7470.773757] LNet: Removed LNI 192.168.204.134@tcp [ 7471.880810] Key type .llcrypt unregistered [ 7471.882867] Key type ._llcrypt unregistered [ 7492.699844] Key type ._llcrypt registered [ 7492.704540] Key type .llcrypt registered [ 7492.818797] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 7494.054933] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7494.198097] alg: No test for adler32 (adler32-zlib) [ 7495.250393] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 7495.545139] LNet: Added LNI 192.168.204.134@tcp [8/256/0/180] [ 7497.303457] Key type lgssc registered [ 7498.568739] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7507.569564] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7507.587427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7509.192258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7515.718485] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 7519.714616] LustreError: 219191:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7519.730429] LustreError: 219191:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7524.833140] LustreError: 219190:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7525.498039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7525.950464] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7530.431582] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 7538.876032] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7545.683513] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7553.961245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7554.538811] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7559.686856] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 7561.365371] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 7570.384432] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7581.577094] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7581.754485] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7582.005475] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7590.698400] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 7599.823607] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7602.088893] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7614.201908] Lustre: Permanently deactivating lustre-OST0001 [ 7614.231866] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7623.197196] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 7636.059688] Lustre: Permanently reactivating lustre-OST0001 [ 7636.078911] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7636.085580] Lustre: Skipped 2 previous similar messages [ 7641.664718] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7641.686637] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7641.694627] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7641.706389] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7641.771454] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7645.673324] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7645.683748] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7645.692446] Lustre: Skipped 2 previous similar messages [ 7645.712591] Lustre: Skipped 1 previous similar message [ 7650.787601] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7650.797647] Lustre: Skipped 1 previous similar message [ 7651.381190] Lustre: server umount lustre-OST0001 complete [ 7655.912340] LustreError: 222748:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7656.929664] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7656.938618] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7656.956903] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7657.956170] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7661.041501] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7661.043408] LustreError: 222748:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7661.046208] Lustre: Skipped 2 previous similar messages [ 7661.076506] LustreError: 222748:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7666.147380] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7666.153456] LustreError: 222748:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7666.159037] Lustre: Skipped 1 previous similar message [ 7666.177358] LustreError: 222748:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7670.244337] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7670.402841] Lustre: server umount lustre-OST0000 complete [ 7676.385647] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7676.393605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7676.401436] Lustre: Skipped 2 previous similar messages [ 7676.418423] Lustre: Skipped 1 previous similar message [ 7681.138876] Lustre: server umount lustre-MDT0000 complete [ 7681.518112] LustreError: 222749:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7681.530045] LustreError: 222749:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7685.445937] LustreError: 222029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789582931 with bad export cookie 14646763141264932699 [ 7685.448579] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7685.453853] LustreError: 222029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7686.105850] Lustre: server umount lustre-MDT0001 complete [ 7697.827909] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [ 7701.116479] Key type lgssc unregistered [ 7701.493703] LNet: 224287:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7701.505310] LNetError: 224287:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7701.519519] LNet: Removed LNI 192.168.204.134@tcp [ 7702.693287] Key type .llcrypt unregistered [ 7702.695612] Key type ._llcrypt unregistered [ 7746.336415] Key type ._llcrypt registered [ 7746.338665] Key type .llcrypt registered [ 7746.575263] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 7747.943078] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7747.967813] alg: No test for adler32 (adler32-zlib) [ 7749.174859] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 7749.492376] LNet: Added LNI 192.168.204.134@tcp [8/256/0/180] [ 7751.180977] Key type lgssc registered [ 7752.607280] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7761.028969] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7761.046906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7762.351531] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7762.381205] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7762.617158] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7767.737592] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 7777.301679] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7777.486944] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7777.553115] Lustre: 226490:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7778.029894] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7783.435723] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 7792.542615] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7799.256242] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7807.372705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7807.760952] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7807.963163] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7813.321276] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 7815.232828] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 7824.514906] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7828.459859] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7836.515399] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7843.691287] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7849.606941] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7865.370450] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7865.376388] Lustre: Skipped 1 previous similar message [ 7882.343786] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7882.347561] Lustre: Skipped 1 previous similar message [ 7917.302495] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7917.314230] Lustre: Skipped 4 previous similar messages [ 7987.661953] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 7987.672348] Lustre: Skipped 8 previous similar messages [ 7997.929892] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7997.939787] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7997.949565] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7997.956230] LustreError: Skipped 1 previous similar message [ 7997.969850] Lustre: Skipped 1 previous similar message [ 8001.636210] Lustre: server umount lustre-OST0000 complete [ 8008.166827] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8008.188598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8008.198484] Lustre: Skipped 1 previous similar message [ 8013.133632] Lustre: server umount lustre-MDT0000 complete [ 8017.847769] LustreError: 229120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789583263 with bad export cookie 2140333869829288159 [ 8017.862755] LustreError: 229120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8017.866040] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8018.373555] Lustre: server umount lustre-MDT0001 complete [ 8032.257585] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [ 8035.786546] Key type lgssc unregistered [ 8036.170824] LNet: 231272:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8036.186907] LNetError: 231272:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8036.209277] LNet: Removed LNI 192.168.204.134@tcp [ 8037.231523] Key type .llcrypt unregistered [ 8037.235591] Key type ._llcrypt unregistered [ 8067.683494] Key type ._llcrypt registered [ 8067.689477] Key type .llcrypt registered [ 8067.867919] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 8069.597856] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8069.696284] alg: No test for adler32 (adler32-zlib) [ 8070.852919] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 8071.368089] LNet: Added LNI 192.168.204.134@tcp [8/256/0/180] [ 8073.208231] Key type lgssc registered [ 8074.889893] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8083.848472] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8083.909263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8085.382110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8091.451546] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 8095.714273] LustreError: 232681:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8095.733467] LustreError: 232681:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8100.839751] LustreError: 232682:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8102.016870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8102.625328] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8108.282279] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 8119.359834] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8128.830198] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8136.739731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8137.305063] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8145.160648] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug -1 all [ 8154.372319] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8155.766720] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 8174.570383] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8174.592714] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8174.628145] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8176.612208] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8176.613970] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8176.614422] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8176.622401] Lustre: Skipped 1 previous similar message [ 8181.732676] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8181.744267] Lustre: Skipped 1 previous similar message [ 8186.858684] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8186.876705] Lustre: Skipped 1 previous similar message [ 8187.361681] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8187.875302] Lustre: server umount lustre-OST0000 complete [ 8195.043881] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8195.060647] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8195.078115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8197.096585] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8199.616241] Lustre: server umount lustre-MDT0000 complete [ 8202.211408] LustreError: 232682:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8202.232514] LustreError: 232682:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8204.235376] LustreError: 232661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789583450 with bad export cookie 4200242189776995600 [ 8204.241154] LustreError: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8204.245853] LustreError: 232661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8204.687849] Lustre: server umount lustre-MDT0001 complete [ 8219.958417] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [ 8224.674558] Key type lgssc unregistered [ 8225.071718] LNet: 236378:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8225.079949] LNetError: 236378:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8225.098446] LNet: Removed LNI 192.168.204.134@tcp [ 8226.582306] Key type .llcrypt unregistered [ 8226.584127] Key type ._llcrypt unregistered [ 8254.831460] Key type ._llcrypt registered [ 8254.837477] Key type .llcrypt registered [ 8254.951626] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 8257.607076] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8257.772302] alg: No test for adler32 (adler32-zlib) [ 8259.162137] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 8259.616345] LNet: Added LNI 192.168.204.134@tcp [8/256/0/180] [ 8261.439324] Key type lgssc registered [ 8263.269383] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8538.752296] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [ 8542.072294] Key type lgssc unregistered [ 8542.411775] LNet: 238402:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8542.417942] LNetError: 238402:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8542.441270] LNet: Removed LNI 192.168.204.134@tcp [ 8543.369404] Key type .llcrypt unregistered [ 8543.371233] Key type ._llcrypt unregistered [ 8578.057441] Key type ._llcrypt registered [ 8578.063748] Key type .llcrypt registered [ 8578.205868] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_hostid [ 8596.711538] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 8597.889057] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8597.929752] alg: No test for adler32 (adler32-zlib) [ 8599.071780] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 8599.531178] LNet: Added LNI 192.168.204.134@tcp [8/256/0/180] [ 8601.330274] Key type lgssc registered [ 8602.681414] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8642.577848] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 8336 sec ========== 14:38:07 (1789583887) [ 8644.716849] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 14:38:09 (1789583889) === [ 8648.463396] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 14:38:13 (1789583893) === [ 8692.847339] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [ 8695.982501] Key type lgssc unregistered [ 8696.387964] LNet: 243643:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8696.402778] LNetError: 243643:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8696.435769] LNet: Removed LNI 192.168.204.134@tcp [ 8697.295352] Key type .llcrypt unregistered [ 8697.297169] Key type ._llcrypt unregistered