[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 487605826 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003140] x2apic enabled [ 0.005004] Switched APIC routing to physical x2apic. [ 0.006015] kvm-guest: setup PV IPIs [ 0.009000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010012] pid_max: default: 32768 minimum: 301 [ 0.011132] LSM: Security Framework initializing [ 0.012050] Yama: becoming mindful. [ 0.013044] SELinux: Initializing. [ 0.014067] *** VALIDATE selinux *** [ 0.022156] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027854] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028162] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029128] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030113] *** VALIDATE tmpfs *** [ 0.031478] *** VALIDATE proc *** [ 0.033063] *** VALIDATE cgroup *** [ 0.034010] *** VALIDATE cgroup2 *** [ 0.035249] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036136] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038028] Spectre V2 : User space: Vulnerable [ 0.039007] Speculative Store Bypass: Vulnerable [ 0.041896] debug: unmapping init [mem 0xffffffff89459000-0xffffffff89460fff] [ 0.043159] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044675] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045020] ... version: 2 [ 0.045942] ... bit width: 48 [ 0.046014] ... generic registers: 4 [ 0.047011] ... value mask: 0000ffffffffffff [ 0.048009] ... max period: 00007fffffffffff [ 0.049012] ... fixed-purpose events: 3 [ 0.049941] ... event mask: 000000070000000f [ 0.050288] rcu: Hierarchical SRCU implementation. [ 0.052399] smp: Bringing up secondary CPUs ... [ 0.053569] x86: Booting SMP configuration: [ 0.054021] .... node #0, CPUs: #1 #2 #3 [ 0.058590] smp: Brought up 1 node, 4 CPUs [ 0.060012] smpboot: Max logical packages: 1 [ 0.061011] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.122043] node 0 deferred pages initialised in 58ms [ 0.126117] devtmpfs: initialized [ 0.128266] x86/mm: Memory block size: 128MB [ 0.131902] gcov: version magic: 0x41383552 [ 0.135242] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.136077] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.137247] pinctrl core: initialized pinctrl subsystem [ 0.138171] [ 0.138793] ************************************************************* [ 0.139012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.140012] ** ** [ 0.141014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.142012] ** ** [ 0.143012] ** This means that this kernel is built to expose internal ** [ 0.144013] ** IOMMU data structures, which may compromise security on ** [ 0.145011] ** your system. ** [ 0.146013] ** ** [ 0.147017] ** If you see this message and you are not debugging the ** [ 0.148013] ** kernel, report this immediately to your vendor! ** [ 0.149014] ** ** [ 0.150014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.151012] ************************************************************* [ 0.152490] NET: Registered protocol family 16 [ 0.154462] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.158074] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.161056] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.165031] cpuidle: using governor menu [ 0.166962] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.169466] PCI: Using configuration type 1 for base access [ 0.171185] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.181068] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.183016] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.187087] cryptd: max_cpu_qlen set to 1000 [ 0.191255] ACPI: Added _OSI(Module Device) [ 0.192013] ACPI: Added _OSI(Processor Device) [ 0.194010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.195012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.200204] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.205399] ACPI: Interpreter enabled [ 0.207053] ACPI: PM: (supports S0 S3 S4 S5) [ 0.208008] ACPI: Using IOAPIC for interrupt routing [ 0.210083] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.212356] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.222419] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.225029] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.228024] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.230061] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.235228] acpiphp: Slot [2] registered [ 0.236096] acpiphp: Slot [5] registered [ 0.237107] acpiphp: Slot [6] registered [ 0.238083] acpiphp: Slot [7] registered [ 0.240104] acpiphp: Slot [8] registered [ 0.241088] acpiphp: Slot [9] registered [ 0.242089] acpiphp: Slot [10] registered [ 0.243084] acpiphp: Slot [3] registered [ 0.244076] acpiphp: Slot [4] registered [ 0.245085] acpiphp: Slot [11] registered [ 0.246094] acpiphp: Slot [12] registered [ 0.248117] acpiphp: Slot [13] registered [ 0.250111] acpiphp: Slot [14] registered [ 0.251088] acpiphp: Slot [15] registered [ 0.253091] acpiphp: Slot [16] registered [ 0.254100] acpiphp: Slot [17] registered [ 0.256111] acpiphp: Slot [18] registered [ 0.257092] acpiphp: Slot [19] registered [ 0.259109] acpiphp: Slot [20] registered [ 0.260071] acpiphp: Slot [21] registered [ 0.262088] acpiphp: Slot [22] registered [ 0.263093] acpiphp: Slot [23] registered [ 0.264090] acpiphp: Slot [24] registered [ 0.265075] acpiphp: Slot [25] registered [ 0.266291] acpiphp: Slot [26] registered [ 0.267096] acpiphp: Slot [27] registered [ 0.269099] acpiphp: Slot [28] registered [ 0.270142] acpiphp: Slot [29] registered [ 0.271087] acpiphp: Slot [30] registered [ 0.273102] acpiphp: Slot [31] registered [ 0.275068] PCI host bridge to bus 0000:00 [ 0.277021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.279042] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.282022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.284022] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.287024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.290045] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.292184] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.293873] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.297405] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.308016] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.313595] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.316020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.318016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.320026] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.323592] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.326862] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.329042] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.332904] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.338015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.350000] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.354014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.360869] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.368014] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.377016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.402015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.413000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.426026] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.433015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.448014] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.459221] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.467014] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.479014] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.500013] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.511213] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.517013] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.525015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.544015] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.557870] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.571015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.581016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.607015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.625255] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.639014] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.649014] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.676016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.690514] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.693366] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.696366] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.699375] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.701717] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.706143] iommu: Default domain type: Passthrough [ 0.708348] SCSI subsystem initialized [ 0.710115] ACPI: bus type USB registered [ 0.711108] usbcore: registered new interface driver usbfs [ 0.713070] usbcore: registered new interface driver hub [ 0.715173] usbcore: registered new device driver usb [ 0.717137] pps_core: LinuxPPS API ver. 1 registered [ 0.718009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.721084] PTP clock support registered [ 0.723124] EDAC MC: Ver: 3.0.0 [ 0.725106] PCI: Using ACPI for IRQ routing [ 0.727752] NetLabel: Initializing [ 0.729010] NetLabel: domain hash size = 128 [ 0.730008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.732070] NetLabel: unlabeled traffic allowed by default [ 0.734117] vgaarb: loaded [ 0.735299] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.737012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.742361] clocksource: Switched to clocksource kvm-clock [ 0.847086] VFS: Disk quotas dquot_6.6.0 [ 0.849286] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.851503] *** VALIDATE ramfs *** [ 0.852685] *** VALIDATE hugetlbfs *** [ 0.854234] pnp: PnP ACPI init [ 0.856355] pnp: PnP ACPI: found 6 devices [ 0.891176] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.896018] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.898535] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.900907] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.903274] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.905902] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.908928] NET: Registered protocol family 2 [ 0.911834] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.917085] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.920796] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.926066] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.929590] TCP: Hash tables configured (established 65536 bind 65536) [ 0.932544] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.935824] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.938814] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.941761] NET: Registered protocol family 1 [ 0.944698] RPC: Registered named UNIX socket transport module. [ 0.947347] RPC: Registered udp transport module. [ 0.948938] RPC: Registered tcp transport module. [ 0.951049] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.953818] NET: Registered protocol family 44 [ 0.955716] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.957940] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.960348] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.962924] PCI: CLS 0 bytes, default 64 [ 0.965411] Unpacking initramfs... [ 2.365514] debug: unmapping init [mem 0xffff93853cc54000-0xffff93853ffbffff] [ 2.372241] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.374566] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.377474] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.901571] Initialise system trusted keyrings [ 2.904296] Key type blacklist registered [ 2.910286] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.920951] zbud: loaded [ 2.924646] *** VALIDATE nfs *** [ 2.926104] *** VALIDATE nfs4 *** [ 2.929324] pstore: using deflate compression [ 2.934433] Platform Keyring initialized [ 3.080667] NET: Registered protocol family 38 [ 3.083071] Key type asymmetric registered [ 3.084721] Asymmetric key parser 'x509' registered [ 3.086804] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.090603] io scheduler mq-deadline registered [ 3.092675] io scheduler kyber registered [ 3.094492] io scheduler bfq registered [ 3.098761] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.102248] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.105355] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.108478] ACPI: Power Button [PWRF] [ 3.114360] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.121660] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.138247] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.152557] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.177494] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.206989] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.237957] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.244799] Non-volatile memory driver v1.3 [ 3.246667] Linux agpgart interface v0.103 [ 3.278559] virtio_blk virtio1: [vda] 149760 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.281730] vda: detected capacity change from 0 to 76677120 [ 3.298618] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.301667] vdb: detected capacity change from 0 to 1073741824 [ 3.318080] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.321085] vdc: detected capacity change from 0 to 2621440000 [ 3.337982] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.343024] vdd: detected capacity change from 0 to 2621440000 [ 3.360083] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.362981] vde: detected capacity change from 0 to 4294967296 [ 3.377443] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.380287] vdf: detected capacity change from 0 to 4294967296 [ 3.386504] libphy: Fixed MDIO Bus: probed [ 3.397541] usbcore: registered new interface driver usbserial_generic [ 3.399934] usbserial: USB Serial support registered for generic [ 3.402392] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.406670] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.408893] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.411834] mousedev: PS/2 mouse device common for all mice [ 3.414812] rtc_cmos 00:05: RTC can wake from S4 [ 3.418165] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.418304] rtc_cmos 00:05: registered as rtc0 [ 3.423223] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.426520] intel_pstate: CPU model not supported [ 3.429361] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.434165] hid: raw HID events driver (C) Jiri Kosina [ 3.436441] usbcore: registered new interface driver usbhid [ 3.438622] usbhid: USB HID core driver [ 3.440209] drop_monitor: Initializing network drop monitor service [ 3.442771] Initializing XFRM netlink socket [ 3.442843] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.444748] NET: Registered protocol family 10 [ 3.450153] Segment Routing with IPv6 [ 3.451512] NET: Registered protocol family 17 [ 3.453785] mpls_gso: MPLS GSO support [ 3.460171] RAS: Correctable Errors collector initialized. [ 3.462166] AVX version of gcm_enc/dec engaged. [ 3.463948] AES CTR mode by8 optimization enabled [ 3.550377] sched_clock: Marking stable (3550352626, 0)->(4545231744, -994879118) [ 3.553930] registered taskstats version 1 [ 3.555761] Loading compiled-in X.509 certificates [ 3.557847] zswap: loaded using pool lzo/zbud [ 3.582752] Key type big_key registered [ 3.595594] Key type encrypted registered [ 3.597148] ima: No TPM chip found, activating TPM-bypass! [ 3.600098] ima: Allocated hash algorithm: sha1 [ 3.601740] ima: No architecture policies found [ 3.603456] evm: Initialising EVM extended attributes: [ 3.605033] evm: security.selinux [ 3.606032] evm: security.ima [ 3.607075] evm: security.capability [ 3.608318] evm: HMAC attrs: 0x1 [ 3.610522] rtc_cmos 00:05: setting system clock to 2026-09-03 12:05:22 UTC (1788437122) [ 3.616772] debug: unmapping init [mem 0xffffffff8a403000-0xffffffff8a5fffff] [ 3.619704] debug: unmapping init [mem 0xffffffff89182000-0xffffffff89458fff] [ 3.628117] Write protecting the kernel read-only data: 28672k [ 3.631196] debug: unmapping init [mem 0xffffffff87803000-0xffffffff879fffff] [ 3.633602] debug: unmapping init [mem 0xffffffff88114000-0xffffffff881fffff] [ 3.668125] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.676315] systemd[1]: Detected virtualization kvm. [ 3.678397] systemd[1]: Detected architecture x86-64. [ 3.680218] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.707829] systemd[1]: No hostname configured. [ 3.709650] systemd[1]: Set hostname to . [ 3.711925] random: systemd: uninitialized urandom read (16 bytes read) [ 3.714645] systemd[1]: Initializing machine ID from random generator. [ 3.766759] random: ln: uninitialized urandom read (6 bytes read) [ 3.858259] random: systemd: uninitialized urandom read (16 bytes read) [ 3.862930] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.868560] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.879215] systemd[1]: Started Memstrack Anylazing Service. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Timers. [ OK ] Listening on udev Control Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Swap. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.527858] device-mapper: uevent: version 1.0.3 [ 4.530867] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.262946] virtio_net virtio0 ens2: renamed from eth0 [ 5.272794] random: fast init done [ 5.321500] scsi host0: ata_piix [ 5.333332] scsi host1: ata_piix [ 5.335320] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.338627] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.937902] dracut-initqueue[579]: RTNETLINK answers: File exists [ 10.113328] random: crng init done [ 10.115437] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.533574] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.714212] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.981704] SELinux: Disabled at runtime. [ 12.041774] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.049960] systemd[1]: Detected virtualization kvm. [ 12.052070] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.530576] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.534931] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.541505] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.545807] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.549394] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.557500] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.562491] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Process Core Dump Socket. Activating swap /dev/disk/by-label/SWAP... Mounting Kernel Debug File System... Mounting Huge Pages File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Control Socket. [ OK [ 12.618155] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS ] Started Dispatch Password Requests to Console Directory Watch. Mounting POSIX Message Queue File System... Starting Apply Kernel Variables... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-getty.slice. Starting udev Coldplug all Devices... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Remount Root and Kernel File Systems... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ 12.947769] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /mnt. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.283411] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.309252] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.394934] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.416239] EDAC sbridge: Ver: 1.1.2 [ 15.053498] Key type dns_resolver registered [ 15.349551] NFS: Registering the id_resolver key type [ 15.352956] Key type id_resolver registered [ 15.354533] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg617-server login: [ 41.354147] libcfs: loading out-of-tree module taints kernel. [ 41.375482] Key type ._llcrypt registered [ 41.377299] Key type .llcrypt registered [ 41.427037] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [ 50.199499] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 50.823682] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 50.831863] alg: No test for adler32 (adler32-zlib) [ 52.091288] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 52.951189] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 54.727150] Key type lgssc registered [ 56.297490] Lustre: Echo OBD driver; http://www.lustre.org/ [ 70.210617] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 109.991973] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 122.782544] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 122.820949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 124.150944] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 124.222281] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 124.428195] Lustre: lustre-MDT0000: new disk, initializing [ 124.600596] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 124.627281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 129.686448] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 141.081455] hrtimer: interrupt took 8464631 ns [ 143.807093] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 143.901164] Lustre: 6481:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 143.934574] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 143.938187] Lustre: Skipped 1 previous similar message [ 143.997305] Lustre: lustre-MDT0001: new disk, initializing [ 144.052833] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 144.076245] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 144.088709] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 148.749809] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 153.512954] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 162.772311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 162.983326] Lustre: lustre-OST0000: new disk, initializing [ 162.986937] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 162.992635] Lustre: 8418:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 163.052850] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 163.569308] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 163.580400] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 163.655060] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 169.094194] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 183.632244] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 183.813240] Lustre: lustre-OST0001: new disk, initializing [ 183.820369] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 183.829674] Lustre: 9490:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 183.912044] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 190.222538] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 193.032089] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 193.040973] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 193.122218] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 202.088990] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 209.424329] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 215.478252] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing check_logdir /tmp/testlogs/ [ 220.540460] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing yml_node [ 225.417795] Lustre: DEBUG MARKER: Client: 2.17.58.2 [ 227.791385] Lustre: DEBUG MARKER: MDS: 2.17.58.2 [ 230.575934] Lustre: DEBUG MARKER: OSS: 2.17.58.2 [ 232.456611] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 3 08:09:10 EDT 2026 [ 250.594533] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 252.282901] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 269.791894] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 269.813198] Lustre: Skipped 1 previous similar message [ 269.815040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 269.832369] Lustre: Skipped 3 previous similar messages [ 274.783765] Lustre: server umount lustre-MDT0000 complete [ 278.543690] LustreError: 6473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788437397 with bad export cookie 13568889037698931275 [ 278.552884] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 278.556450] LustreError: 6473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 280.032283] LustreError: 8979:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 280.035399] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 280.055379] LustreError: 8979:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 280.088136] Lustre: Skipped 2 previous similar messages [ 280.097722] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 280.109097] Lustre: Skipped 1 previous similar message [ 283.039950] LustreError: 12123:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 283.041177] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 283.052985] LustreError: 12123:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 283.063805] Lustre: Skipped 1 previous similar message [ 284.926272] Lustre: server umount lustre-MDT0001 complete [ 288.493769] Lustre: server umount lustre-OST0000 complete [ 292.264267] Lustre: server umount lustre-OST0001 complete [ 299.330876] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [ 307.581441] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 339.020645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 339.256161] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 339.276797] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 339.338556] Lustre: lustre-MDT0000: new disk, initializing [ 339.397758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 339.415540] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 343.473831] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 353.455647] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 353.583177] Lustre: 15041:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 353.664328] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 353.669652] Lustre: Skipped 1 previous similar message [ 353.770645] Lustre: lustre-MDT0001: new disk, initializing [ 353.910475] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 353.950790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 353.964452] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 359.103530] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 369.376287] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 375.996868] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 382.754889] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 383.035357] Lustre: lustre-OST0000: new disk, initializing [ 383.040904] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 383.051248] Lustre: 17015:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 383.143735] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 384.179645] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 384.190703] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 384.352169] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 388.840811] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 397.968975] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 403.676894] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 404.065233] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 408.026975] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 408.220423] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 415.208634] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 415.211677] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 415.218266] LustreError: Skipped 1 previous similar message [ 415.226534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 415.235227] Lustre: Skipped 2 previous similar messages [ 415.256412] Lustre: Skipped 1 previous similar message [ 416.774725] Lustre: server umount lustre-OST0000 complete [ 425.441372] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 425.449230] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 425.458905] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 425.465595] Lustre: Skipped 1 previous similar message [ 426.791837] Lustre: server umount lustre-MDT0000 complete [ 430.561499] LustreError: 16475:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 430.582403] LustreError: 16475:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 430.595682] LustreError: 15033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788437549 with bad export cookie 13568889037698932808 [ 430.604742] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 430.608605] LustreError: 15033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 431.043584] Lustre: server umount lustre-MDT0001 complete [ 434.977183] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 08:12:32 (1788437552) [ 441.322316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 441.610155] LustreError: 19281:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 441.672898] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 445.463504] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 446.954852] LustreError: 19282:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 453.992938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 459.082785] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 466.954441] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 473.211190] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 480.670609] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 487.953764] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 495.445749] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 506.848264] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 506.859961] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 506.885497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 506.889449] Lustre: Skipped 1 previous similar message [ 509.925274] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 509.935348] LustreError: Skipped 1 previous similar message [ 512.272512] Lustre: server umount lustre-OST0000 complete [ 520.160713] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 520.181736] Lustre: Skipped 1 previous similar message [ 522.595966] Lustre: server umount lustre-MDT0000 complete [ 525.290555] LustreError: 19282:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 525.307182] LustreError: 19282:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 525.889382] LustreError: 19262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788437644 with bad export cookie 13568889037698933690 [ 525.900501] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 525.901225] LustreError: 19262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 526.212773] Lustre: server umount lustre-MDT0001 complete [ 536.485479] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 539.091922] Key type lgssc unregistered [ 539.448763] LNet: 22813:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 539.463920] LNetError: 22813:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 539.497696] LNet: Removed LNI 192.168.206.117@tcp [ 540.662250] Key type .llcrypt unregistered [ 540.667297] Key type ._llcrypt unregistered [ 561.899802] Key type ._llcrypt registered [ 561.903474] Key type .llcrypt registered [ 562.043162] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 563.157086] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 563.360235] alg: No test for adler32 (adler32-zlib) [ 564.488782] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 564.788072] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 566.463641] Key type lgssc registered [ 567.525561] Lustre: Echo OBD driver; http://www.lustre.org/ [ 575.028485] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 575.051098] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 576.457115] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 579.907599] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 586.720923] LustreError: 24202:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 586.739904] LustreError: 24202:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 587.824459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 588.192274] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 591.938539] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 598.421836] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 603.205583] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 609.742950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 610.032693] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 615.297486] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 621.310232] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 621.552131] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 636.896787] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 636.907996] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 636.911359] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 636.923761] Lustre: Skipped 1 previous similar message [ 636.933049] Lustre: Skipped 1 previous similar message [ 637.375213] Lustre: server umount lustre-OST0000 complete [ 642.022826] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 642.044291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 642.052458] Lustre: Skipped 1 previous similar message [ 646.543899] Lustre: server umount lustre-MDT0000 complete [ 647.140630] LustreError: 25268:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 647.167612] LustreError: 25268:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 650.415661] LustreError: 26046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788437769 with bad export cookie 14755656371602267070 [ 650.417054] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 650.433790] LustreError: 26046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 650.688935] Lustre: server umount lustre-MDT0001 complete [ 661.744686] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 665.086727] Key type lgssc unregistered [ 665.451347] LNet: 28176:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 665.463800] LNetError: 28176:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 665.501905] LNet: Removed LNI 192.168.206.117@tcp [ 666.398930] Key type .llcrypt unregistered [ 666.401285] Key type ._llcrypt unregistered [ 688.192655] Key type ._llcrypt registered [ 688.194619] Key type .llcrypt registered [ 688.321377] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 689.369615] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 689.449136] alg: No test for adler32 (adler32-zlib) [ 690.534862] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 690.812338] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 692.519474] Key type lgssc registered [ 693.868882] Lustre: Echo OBD driver; http://www.lustre.org/ [ 701.386814] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 701.398739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 702.810226] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 706.991121] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 712.175860] LustreError: 29565:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 712.195023] LustreError: 29565:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 714.839395] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 715.210409] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 719.155423] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 725.501862] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 730.098281] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 742.279246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 742.535795] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 747.690490] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 747.980673] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 754.764538] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 767.972795] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 767.988499] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 768.011632] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 770.532313] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 770.544840] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 770.551529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 770.568992] Lustre: Skipped 1 previous similar message [ 772.344094] Lustre: server umount lustre-OST0000 complete [ 776.672323] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 776.677588] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 776.693278] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 780.768969] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 780.774254] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 782.072036] Lustre: server umount lustre-MDT0000 complete [ 785.294101] LustreError: 29547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788437904 with bad export cookie 16146754434302208857 [ 785.304582] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 785.622716] Lustre: server umount lustre-MDT0001 complete [ 795.704534] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 798.171565] Key type lgssc unregistered [ 798.449106] LNet: 33441:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 798.457162] LNetError: 33441:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 798.471648] LNet: Removed LNI 192.168.206.117@tcp [ 799.152514] Key type .llcrypt unregistered [ 799.154552] Key type ._llcrypt unregistered [ 821.877084] Key type ._llcrypt registered [ 821.878637] Key type .llcrypt registered [ 822.023789] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 823.510811] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 823.605254] alg: No test for adler32 (adler32-zlib) [ 824.853752] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 825.109461] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 826.783199] Key type lgssc registered [ 828.068857] Lustre: Echo OBD driver; http://www.lustre.org/ [ 835.844569] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 835.861658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 837.228699] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 841.705473] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 847.841639] LustreError: 34851:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 847.861984] LustreError: 34851:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 850.365467] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 850.830376] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 855.626962] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 862.614757] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 868.032500] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 875.036734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 875.430630] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 880.893596] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 882.624450] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 887.157812] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 896.993532] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 897.015834] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 897.035437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 898.032902] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 898.035536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 898.035552] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 898.062416] Lustre: Skipped 1 previous similar message [ 903.136962] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 908.257762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 908.270526] Lustre: Skipped 2 previous similar messages [ 911.327121] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 911.510854] Lustre: server umount lustre-OST0000 complete [ 917.474240] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 917.478345] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 917.491693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 922.450542] Lustre: server umount lustre-MDT0000 complete [ 923.616874] LustreError: 34846:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 923.632588] LustreError: 34846:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 927.284953] LustreError: 34831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438046 with bad export cookie 5767067330838492302 [ 927.292617] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 927.299599] LustreError: 34831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 927.726369] Lustre: server umount lustre-MDT0001 complete [ 938.387611] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 940.628056] Key type lgssc unregistered [ 940.952607] LNet: 38383:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 940.957818] LNetError: 38383:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 940.975131] LNet: Removed LNI 192.168.206.117@tcp [ 941.648669] Key type .llcrypt unregistered [ 941.653226] Key type ._llcrypt unregistered [ 963.529431] Key type ._llcrypt registered [ 963.532287] Key type .llcrypt registered [ 963.634118] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 964.644140] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 964.750717] alg: No test for adler32 (adler32-zlib) [ 965.772850] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 966.024624] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 967.687174] Key type lgssc registered [ 968.671874] Lustre: Echo OBD driver; http://www.lustre.org/ [ 975.911800] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 975.923425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 977.413151] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 982.040789] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 987.623529] LustreError: 39789:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 987.641840] LustreError: 39789:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 990.282242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 990.581066] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 994.746722] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 1002.327573] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1008.240624] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1014.061516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1014.433124] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1019.403881] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 1022.682944] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1026.908163] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1033.196064] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1033.197436] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1033.205365] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1033.227229] Lustre: Skipped 1 previous similar message [ 1033.242377] Lustre: Skipped 1 previous similar message [ 1034.251062] Lustre: lustre-OST0000: Not available for connect from 192.168.206.17@tcp (stopping) [ 1038.306843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1038.319028] Lustre: Skipped 1 previous similar message [ 1038.826230] Lustre: server umount lustre-OST0000 complete [ 1058.786985] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1058.793119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1058.810352] Lustre: Skipped 1 previous similar message [ 1058.826886] Lustre: Skipped 1 previous similar message [ 1063.913312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1063.922920] Lustre: Skipped 1 previous similar message [ 1064.724354] Lustre: server umount lustre-MDT0000 complete [ 1064.929606] LustreError: 39790:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1064.938819] LustreError: 39790:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1068.341817] LustreError: 39769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438187 with bad export cookie 7820584169008858607 [ 1068.345210] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1068.353328] LustreError: 39769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1075.051625] Lustre: server umount lustre-MDT0001 complete [ 1085.571350] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 1088.616811] Key type lgssc unregistered [ 1088.901535] LNet: 43497:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1088.911517] LNetError: 43497:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1088.942244] LNet: Removed LNI 192.168.206.117@tcp [ 1090.060739] Key type .llcrypt unregistered [ 1090.063546] Key type ._llcrypt unregistered [ 1112.457509] Key type ._llcrypt registered [ 1112.459573] Key type .llcrypt registered [ 1112.546073] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 1113.662428] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1113.700491] alg: No test for adler32 (adler32-zlib) [ 1114.842165] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1115.112854] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 1116.863267] Key type lgssc registered [ 1118.209738] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1127.329739] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1127.347086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1129.007025] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1134.494300] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 1138.147782] LustreError: 44904:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1138.156673] LustreError: 44904:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1143.272310] LustreError: 44905:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1143.540702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1144.071905] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1148.930772] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 1156.412238] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1162.238275] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1169.331195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1169.792808] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1173.872102] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1175.987154] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 1183.256929] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1193.441601] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1193.443125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1193.454198] Lustre: Skipped 2 previous similar messages [ 1193.470159] Lustre: Skipped 2 previous similar messages [ 1196.040294] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.17@tcp (stopping) [ 1198.561494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1198.576252] Lustre: Skipped 2 previous similar messages [ 1199.063939] Lustre: server umount lustre-MDT0000 complete [ 1201.157821] LustreError: 44900:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1202.432328] LustreError: 44884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438321 with bad export cookie 9258034944883909790 [ 1202.440694] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1202.443907] LustreError: 44884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1203.680369] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1203.681447] LustreError: 44899:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1203.706490] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1207.712679] LustreError: 46974:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1207.714658] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1207.722404] LustreError: 46974:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1207.725988] Lustre: Skipped 1 previous similar message [ 1208.803223] Lustre: server umount lustre-MDT0001 complete [ 1230.323365] Lustre: server umount lustre-OST0000 complete [ 1247.954744] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 1250.318432] Key type lgssc unregistered [ 1250.586873] LNet: 48765:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1250.594334] LNetError: 48765:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1250.611156] LNet: Removed LNI 192.168.206.117@tcp [ 1251.425343] Key type .llcrypt unregistered [ 1251.427337] Key type ._llcrypt unregistered [ 1274.742915] Key type ._llcrypt registered [ 1274.744432] Key type .llcrypt registered [ 1274.845587] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 1275.739957] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1275.796125] alg: No test for adler32 (adler32-zlib) [ 1277.028480] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1277.448578] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 1279.143206] Key type lgssc registered [ 1280.356656] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1288.078613] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1288.093336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1303.583578] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1308.703342] LustreError: 50148:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.117@tcp: failed processing log, type 4: rc = -110 [ 1335.327201] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1341.675363] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 1348.857709] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1516.885882] Lustre: server umount lustre-OST0000 complete [ 1534.599694] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 1537.251871] Key type lgssc unregistered [ 1537.468384] LNet: 51822:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1537.477290] LNetError: 51822:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1537.493656] LNet: Removed LNI 192.168.206.117@tcp [ 1538.117848] Key type .llcrypt unregistered [ 1538.125882] Key type ._llcrypt unregistered [ 1560.371958] Key type ._llcrypt registered [ 1560.379030] Key type .llcrypt registered [ 1560.517964] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 1561.359592] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1561.476472] alg: No test for adler32 (adler32-zlib) [ 1562.624544] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1562.978906] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 1564.751489] Key type lgssc registered [ 1565.917851] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1574.107863] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1574.131706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1575.609752] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1580.068263] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 1585.127114] LustreError: 53237:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1585.155110] LustreError: 53237:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1588.722585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1589.007685] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1593.582977] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 1601.714736] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1608.168878] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1614.954102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1615.456975] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1620.533209] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1621.662180] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 1628.943570] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1634.020390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1634.021132] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1637.259405] Lustre: server umount lustre-OST0000 complete [ 1644.513019] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1644.517077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1644.535518] Lustre: Skipped 1 previous similar message [ 1644.552886] Lustre: Skipped 2 previous similar messages [ 1646.758753] Lustre: server umount lustre-MDT0000 complete [ 1649.635357] LustreError: 53966:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1649.650815] LustreError: 53966:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1650.108865] LustreError: 53217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438768 with bad export cookie 9365754936740279368 [ 1650.118241] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1650.125583] LustreError: 53217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1650.480550] Lustre: server umount lustre-MDT0001 complete [ 1660.552545] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 1663.305887] Key type lgssc unregistered [ 1663.595061] LNet: 56766:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1663.599235] LNetError: 56766:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1663.626486] LNet: Removed LNI 192.168.206.117@tcp [ 1664.505912] Key type .llcrypt unregistered [ 1664.508212] Key type ._llcrypt unregistered [ 1688.155290] Key type ._llcrypt registered [ 1688.159204] Key type .llcrypt registered [ 1688.345766] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 1689.353315] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1689.381676] alg: No test for adler32 (adler32-zlib) [ 1690.450134] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1690.893641] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 1692.599165] Key type lgssc registered [ 1693.813273] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1702.215519] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1702.231930] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1717.791607] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1722.911314] LustreError: 58150:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.117@tcp: failed processing log, type 4: rc = -110 [ 1749.599271] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1755.237432] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 1761.167565] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1767.325106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1767.937059] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1768.026236] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1771.740414] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 1773.033711] LustreError: 58175:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1773.066525] LustreError: 58175:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1778.150193] LustreError: 58176:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1779.650640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1780.040376] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1783.995559] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 1791.588677] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1797.690621] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1800.681491] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1800.687522] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1800.713986] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1803.772835] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1803.778392] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1803.786266] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1806.263294] Lustre: server umount lustre-OST0000 complete [ 1831.392499] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1831.403433] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1831.430711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1831.434368] Lustre: Skipped 2 previous similar messages [ 1834.468459] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1834.474503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1834.488051] Lustre: Skipped 1 previous similar message [ 1835.804252] Lustre: server umount lustre-MDT0000 complete [ 1839.587942] LustreError: 59185:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1839.617577] LustreError: 59185:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1839.924800] LustreError: 58158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438958 with bad export cookie 6343678113917201303 [ 1839.936022] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1839.937808] LustreError: 58158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1846.269676] Lustre: server umount lustre-MDT0001 complete [ 1859.047902] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 1862.440542] Key type lgssc unregistered [ 1862.728419] LNet: 61893:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1862.734268] LNetError: 61893:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1863.783728] LNet: Removed LNI 192.168.206.117@tcp [ 1864.700358] Key type .llcrypt unregistered [ 1864.701886] Key type ._llcrypt unregistered [ 1889.319988] Key type ._llcrypt registered [ 1889.323602] Key type .llcrypt registered [ 1889.431202] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 1890.664436] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1890.710402] alg: No test for adler32 (adler32-zlib) [ 1891.860585] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1892.100365] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 1893.783164] Key type lgssc registered [ 1894.844458] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1902.817473] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1902.838230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1904.210231] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1908.631585] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 1914.343983] LustreError: 63301:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1914.375071] LustreError: 63301:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1919.229907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1919.457597] LustreError: 63302:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1919.891299] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1924.773365] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 1932.851677] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1939.672292] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1946.952950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1947.274329] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1949.429123] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1953.239521] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 1961.467782] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1979.075946] Lustre: server umount lustre-OST0000 complete [ 1985.507740] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1985.515155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1985.525270] Lustre: Skipped 1 previous similar message [ 1990.625191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1990.638516] Lustre: Skipped 2 previous similar messages [ 1992.744178] Lustre: server umount lustre-MDT0000 complete [ 1995.753628] LustreError: 63302:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1995.780485] LustreError: 63302:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1996.767420] Lustre: 62613:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788439099/real 1788439099] req@000000003f3b6a00 x1875314223378560/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788439115 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1996.798533] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1996.830808] LustreError: 63283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439115 with bad export cookie 6540207953410262767 [ 1996.834678] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1996.841385] LustreError: 63283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1997.356411] Lustre: server umount lustre-MDT0001 complete [ 2008.640318] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 2011.519195] Key type lgssc unregistered [ 2011.812438] LNet: 66857:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2011.822190] LNetError: 66857:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2011.845928] LNet: Removed LNI 192.168.206.117@tcp [ 2012.973127] Key type .llcrypt unregistered [ 2012.986543] Key type ._llcrypt unregistered [ 2038.495587] Key type ._llcrypt registered [ 2038.498898] Key type .llcrypt registered [ 2038.600748] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 2039.817169] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2039.881746] alg: No test for adler32 (adler32-zlib) [ 2040.994170] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2041.252771] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 2042.927168] Key type lgssc registered [ 2044.095053] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2053.450915] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 08:39:31 (1788439171) [ 2060.727682] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2060.774389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2062.619389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2068.316788] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2073.063690] LustreError: 68558:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2073.090960] LustreError: 68558:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2078.187781] LustreError: 68557:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2079.792671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2080.165042] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2084.898982] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2091.748352] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2098.077720] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2105.068109] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2105.461422] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2107.534615] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2112.281455] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2120.800699] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2129.779711] Lustre: Failing over lustre-MDT0000 [ 2130.188083] Lustre: server umount lustre-MDT0000 complete [ 2130.404469] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2130.406392] LustreError: 69960:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2130.427085] Lustre: Skipped 1 previous similar message [ 2130.454895] LustreError: 69960:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2133.038538] LustreError: 68552:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2138.095078] LustreError: 68554:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2138.131380] LustreError: 68554:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2140.740603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2140.880408] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2141.061207] Lustre: *** cfs_fail_loc=135, val=0*** [ 2141.149516] LustreError: 71645:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2141.155088] LustreError: 71645:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.117@tcp: cfg command failed: rc = -2 [ 2141.161516] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2141.161516] [ 2141.168430] LustreError: MGC192.168.206.117@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2141.179929] LustreError: 71633:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2141.191289] LustreError: 71633:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [ 2141.219330] LustreError: 71633:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 2141.232291] LustreError: 71633:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2141.264485] Lustre: server umount lustre-MDT0000 complete [ 2141.271794] LustreError: 71633:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2148.337714] LustreError: 68553:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2148.369885] LustreError: 68553:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 2148.435271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2148.652263] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2148.981678] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2149.037669] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2149.401274] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2153.274673] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2154.479884] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2154.504697] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2154.567717] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2161.122564] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2161.134098] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2161.150690] Lustre: Skipped 1 previous similar message [ 2161.159960] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2164.709421] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2164.720863] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2166.694405] Lustre: server umount lustre-OST0000 complete [ 2174.946421] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2174.955212] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2174.983164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2174.990484] Lustre: Skipped 2 previous similar messages [ 2176.720744] Lustre: server umount lustre-MDT0000 complete [ 2180.070087] LustreError: 68552:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2180.122242] LustreError: 68552:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2180.812793] LustreError: 68539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439299 with bad export cookie 10396513772323728808 [ 2180.831654] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2181.375692] Lustre: server umount lustre-MDT0001 complete [ 2193.437354] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 2196.670428] Key type lgssc unregistered [ 2196.989619] LNet: 73421:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2197.007752] LNetError: 73421:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2197.026138] LNet: Removed LNI 192.168.206.117@tcp [ 2198.078653] Key type .llcrypt unregistered [ 2198.086281] Key type ._llcrypt unregistered [ 2223.621241] Key type ._llcrypt registered [ 2223.624553] Key type .llcrypt registered [ 2223.811431] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 2225.584304] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2225.992617] alg: No test for adler32 (adler32-zlib) [ 2227.180604] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2227.475798] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 2229.207287] Key type lgssc registered [ 2230.588937] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2239.273185] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2239.311162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2240.834626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2246.011202] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2250.219954] LustreError: 74833:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2250.239234] LustreError: 74833:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2255.330268] LustreError: 74834:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2256.010490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2256.611297] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2261.602950] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2268.903938] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2275.637396] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2282.467725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2282.852543] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2285.946134] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2289.278343] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2297.107962] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2306.857199] Lustre: Failing over lustre-MDT0000 [ 2307.111831] Lustre: server umount lustre-MDT0000 complete [ 2307.554636] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2307.566646] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2307.585795] LustreError: 74834:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2310.158840] LustreError: 76675:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2311.653694] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2315.242552] LustreError: 76675:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2315.258967] LustreError: 76675:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2316.090996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2316.226707] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2316.400389] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2316.509636] LustreError: 77919:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2316.523299] LustreError: 77919:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.117@tcp: cfg command failed: rc = -9 [ 2316.543329] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2316.543329] [ 2316.552795] LustreError: MGC192.168.206.117@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2316.565552] LustreError: 77908:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2316.572579] LustreError: 77908:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -9 [ 2316.586654] LustreError: 77908:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 2316.597014] LustreError: 77908:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2316.626492] Lustre: server umount lustre-MDT0000 complete [ 2316.641435] LustreError: 77908:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2323.677775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2323.884784] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2323.906782] LustreError: 74825:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2324.179683] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2324.227932] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2325.488473] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2328.922883] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2329.594374] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2329.646292] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2329.685939] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2337.266978] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2337.275261] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2337.290654] Lustre: Skipped 1 previous similar message [ 2337.300291] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2339.812070] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2339.813719] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2343.246574] Lustre: server umount lustre-OST0000 complete [ 2350.049595] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2350.076567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2350.082924] Lustre: Skipped 2 previous similar messages [ 2353.689188] Lustre: server umount lustre-MDT0000 complete [ 2355.169159] LustreError: 74833:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2355.193120] LustreError: 74833:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 2357.596818] LustreError: 75568:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439476 with bad export cookie 7033859264999295819 [ 2357.603760] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2357.612666] LustreError: 75568:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2357.954564] Lustre: server umount lustre-MDT0001 complete [ 2370.298357] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 2373.656549] Key type lgssc unregistered [ 2374.031651] LNet: 79702:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2374.035863] LNetError: 79702:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2374.072045] LNet: Removed LNI 192.168.206.117@tcp [ 2375.128667] Key type .llcrypt unregistered [ 2375.131152] Key type ._llcrypt unregistered [ 2388.487546] Key type ._llcrypt registered [ 2388.490266] Key type .llcrypt registered [ 2388.586242] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 2388.850323] Key type .llcrypt unregistered [ 2388.852420] Key type ._llcrypt unregistered [ 2414.468252] Key type ._llcrypt registered [ 2414.470359] Key type .llcrypt registered [ 2414.571102] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 2415.633585] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2415.723895] alg: No test for adler32 (adler32-zlib) [ 2416.890687] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2417.151106] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 2418.887406] Key type lgssc registered [ 2420.359835] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2429.260062] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2429.289147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2430.846646] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2435.765177] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2440.165346] LustreError: 81597:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2440.183493] LustreError: 81597:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2445.295256] LustreError: 81596:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2445.891234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2446.353768] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2451.964105] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2460.795136] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2466.782373] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2474.298453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2474.617118] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2475.630992] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2480.268559] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2487.842681] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2502.625693] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2502.638821] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2502.654040] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2502.701861] Lustre: Skipped 1 previous similar message [ 2507.746657] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2507.752167] Lustre: Skipped 1 previous similar message [ 2508.455619] Lustre: server umount lustre-OST0000 complete [ 2512.865324] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2512.865469] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2512.884697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2512.889840] Lustre: Skipped 1 previous similar message [ 2512.899218] Lustre: Skipped 2 previous similar messages [ 2517.972735] Lustre: server umount lustre-MDT0000 complete [ 2517.991659] LustreError: 83497:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2518.006403] LustreError: 83497:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2521.708339] LustreError: 84399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439640 with bad export cookie 14531172663358865987 [ 2521.709074] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2521.714158] LustreError: 84399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2521.967236] Lustre: server umount lustre-MDT0001 complete [ 2532.575573] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 2535.581458] Key type lgssc unregistered [ 2536.005050] LNet: 85151:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2536.011601] LNetError: 85151:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2537.063301] LNet: Removed LNI 192.168.206.117@tcp [ 2538.027833] Key type .llcrypt unregistered [ 2538.030347] Key type ._llcrypt unregistered [ 2564.222775] Key type ._llcrypt registered [ 2564.225284] Key type .llcrypt registered [ 2564.399902] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 2566.348179] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2566.831485] alg: No test for adler32 (adler32-zlib) [ 2568.170264] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2568.680617] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 2570.447237] Key type lgssc registered [ 2571.773795] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2579.302569] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2579.315110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2580.831937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2585.313371] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2590.183326] LustreError: 86560:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2590.201674] LustreError: 86560:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2594.062374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2594.497619] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2599.466893] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2607.341661] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2613.568756] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2620.754423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2621.157053] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2625.260433] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2627.399454] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2634.469408] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2645.995043] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2646.019388] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2646.073714] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2650.081969] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2650.088529] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2650.090455] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2650.111558] Lustre: Skipped 1 previous similar message [ 2652.032471] Lustre: server umount lustre-OST0000 complete [ 2660.321424] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2660.326091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2660.336012] Lustre: Skipped 1 previous similar message [ 2662.646852] Lustre: server umount lustre-MDT0000 complete [ 2665.441482] LustreError: 86557:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2665.467931] LustreError: 86557:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2666.780988] LustreError: 86540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439785 with bad export cookie 7141786816632216206 [ 2666.794816] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2666.800629] LustreError: 86540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2667.117074] Lustre: server umount lustre-MDT0001 complete [ 2677.983823] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 2681.160563] Key type lgssc unregistered [ 2681.601274] LNet: 90093:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2681.616034] LNetError: 90093:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2682.660877] LNet: Removed LNI 192.168.206.117@tcp [ 2683.981608] Key type .llcrypt unregistered [ 2683.983661] Key type ._llcrypt unregistered [ 2709.490828] Key type ._llcrypt registered [ 2709.493619] Key type .llcrypt registered [ 2709.648201] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 2711.191579] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2711.275297] alg: No test for adler32 (adler32-zlib) [ 2712.573677] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2712.994930] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 2714.767241] Key type lgssc registered [ 2715.997656] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2725.305992] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2725.328424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2726.955159] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2732.356333] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2737.122646] LustreError: 91504:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2737.149108] LustreError: 91504:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2741.165269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2741.406299] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2745.982812] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2753.408704] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2759.532643] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2766.730901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2767.197793] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2772.347116] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2773.813843] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2782.449223] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2798.052926] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2798.070871] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2798.095753] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2799.737338] Lustre: server umount lustre-OST0000 complete [ 2807.268334] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2807.280598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2807.282587] Lustre: Skipped 1 previous similar message [ 2807.301548] Lustre: Skipped 1 previous similar message [ 2810.499784] Lustre: server umount lustre-MDT0000 complete [ 2812.391453] LustreError: 91504:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2812.419492] LustreError: 91504:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2814.728085] LustreError: 91485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439933 with bad export cookie 17970036098527659906 [ 2814.730856] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2814.739523] LustreError: 91485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2824.295211] Lustre: server umount lustre-MDT0001 complete [ 2835.486243] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 2838.290260] Key type lgssc unregistered [ 2838.655422] LNet: 95057:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2838.672668] LNetError: 95057:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2839.719327] LNet: Removed LNI 192.168.206.117@tcp [ 2840.574277] Key type .llcrypt unregistered [ 2840.576281] Key type ._llcrypt unregistered [ 2863.290909] Key type ._llcrypt registered [ 2863.293056] Key type .llcrypt registered [ 2863.414188] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 2864.506276] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2864.675299] alg: No test for adler32 (adler32-zlib) [ 2865.811217] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2866.197955] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 2867.992188] Key type lgssc registered [ 2869.195626] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2877.325803] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2877.348821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2892.959412] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2898.079352] LustreError: 96439:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.117@tcp: failed processing log, type 4: rc = -110 [ 2924.703845] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2931.017125] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2939.968222] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2945.662100] Lustre: server umount lustre-OST0000 complete [ 2955.059769] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 08:54:32 (1788440072) [ 2961.506254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2962.330815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2966.955457] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2967.523377] LustreError: 98079:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2967.562779] LustreError: 98079:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2972.640168] LustreError: 98078:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2976.179495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2976.654767] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2982.275688] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2991.395120] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2997.220952] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3005.614612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3006.096720] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3007.177515] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3011.990369] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 3019.654098] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3032.034830] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3032.037524] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3032.062606] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3032.069218] Lustre: Skipped 1 previous similar message [ 3036.237834] Lustre: server umount lustre-OST0000 complete [ 3042.286550] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3042.289961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3042.313289] Lustre: Skipped 2 previous similar messages [ 3046.883859] Lustre: server umount lustre-MDT0000 complete [ 3047.392561] LustreError: 98074:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3047.419867] LustreError: 98074:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3052.183687] LustreError: 98814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440171 with bad export cookie 8044366971608410321 [ 3052.190377] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3052.195887] LustreError: 98814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3052.538602] Lustre: server umount lustre-MDT0001 complete [ 3063.185129] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 3065.968329] Key type lgssc unregistered [ 3066.171689] LNet: 101771:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3066.177193] LNetError: 101771:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3066.189250] LNet: Removed LNI 192.168.206.117@tcp [ 3066.942511] Key type .llcrypt unregistered [ 3066.945795] Key type ._llcrypt unregistered [ 3111.182856] Key type ._llcrypt registered [ 3111.184983] Key type .llcrypt registered [ 3111.297320] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 3112.563581] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3112.605677] alg: No test for adler32 (adler32-zlib) [ 3113.745499] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3114.075976] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 3115.841861] Key type lgssc registered [ 3117.150200] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3125.371826] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3125.408057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3126.947968] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3131.181584] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 3136.483463] LustreError: 103954:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3136.501580] LustreError: 103954:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3139.925296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3140.303918] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3144.735769] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 3153.248274] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3159.555617] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3166.895314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3167.340283] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3171.460501] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3174.041825] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 3182.015648] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3194.850404] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3194.857151] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3194.881530] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3197.409017] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3197.415996] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3197.426415] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3197.462645] Lustre: Skipped 1 previous similar message [ 3199.359200] Lustre: server umount lustre-OST0000 complete [ 3207.136843] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3207.150356] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3207.173778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3212.778234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3212.796042] Lustre: Skipped 2 previous similar messages [ 3217.375250] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3217.613624] Lustre: server umount lustre-MDT0000 complete [ 3217.906128] LustreError: 104684:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3217.930754] LustreError: 104684:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3221.675651] LustreError: 105799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440340 with bad export cookie 14992598961051815029 [ 3221.684370] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3221.686502] LustreError: 105799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3222.085988] Lustre: server umount lustre-MDT0001 complete [ 3234.436813] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 3237.087732] Key type lgssc unregistered [ 3237.362914] LNet: 107490:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3237.371759] LNetError: 107490:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3237.384595] LNet: Removed LNI 192.168.206.117@tcp [ 3238.230486] Key type .llcrypt unregistered [ 3238.236208] Key type ._llcrypt unregistered [ 3257.165516] Key type ._llcrypt registered [ 3257.168979] Key type .llcrypt registered [ 3257.263532] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 3258.460694] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3258.512154] alg: No test for adler32 (adler32-zlib) [ 3259.615465] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3259.893507] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 3261.577562] Key type lgssc registered [ 3262.882950] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3270.594750] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3270.614311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3286.111579] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3291.231323] LustreError: 108729:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.117@tcp: failed processing log, type 4: rc = -110 [ 3317.919239] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3323.423506] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 3331.851463] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3339.438654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3339.766953] LustreError: MGC192.168.206.117@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3339.785812] LustreError: 109753:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3339.805021] LustreError: 109753:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [ 3339.834719] LustreError: 109753:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 3339.987069] Lustre: server umount lustre-MDT0000 complete [ 3339.990809] LustreError: 109753:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3360.223110] Lustre: 108067:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788440463/real 1788440463] req@00000000518b74c5 x1875315657805056/t0(0) o400->MGC192.168.206.117@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788440479 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3360.261643] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3360.287128] LustreError: 108741:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3380.821302] Lustre: server umount lustre-OST0000 complete [ 3391.024744] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [ 3401.584169] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 3436.928642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3437.234262] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3437.273203] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3437.396328] Lustre: lustre-MDT0000: new disk, initializing [ 3437.476176] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3437.494737] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3442.902320] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 3454.606870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3454.759602] Lustre: 112562:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3454.813218] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3454.818273] Lustre: Skipped 1 previous similar message [ 3454.888647] Lustre: lustre-MDT0001: new disk, initializing [ 3455.008216] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3455.046652] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3455.058406] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3459.810823] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 3468.998921] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3476.368849] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3484.598752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3484.942788] Lustre: lustre-OST0000: new disk, initializing [ 3484.952367] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3484.963521] Lustre: 114531:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3485.121211] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3490.860178] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3490.871907] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3491.004842] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3492.258863] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 3502.462886] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3507.708077] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3507.933537] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3512.988653] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3513.252435] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3516.897165] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3516.907512] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3516.934715] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3521.754509] Lustre: server umount lustre-OST0000 complete [ 3526.624814] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3526.634129] LustreError: Skipped 1 previous similar message [ 3526.639697] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3526.658107] Lustre: Skipped 1 previous similar message [ 3526.672323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3526.675124] Lustre: Skipped 1 previous similar message [ 3532.340729] Lustre: server umount lustre-MDT0000 complete [ 3536.212486] LustreError: 114542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440655 with bad export cookie 5854019485244816497 [ 3536.228105] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3536.657266] Lustre: server umount lustre-MDT0001 complete [ 3545.222535] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 09:04:23 (1788440663) [ 3549.004719] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3565.916178] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [ 3576.748855] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 3614.813780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3614.993169] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3615.009651] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3615.064797] Lustre: lustre-MDT0000: new disk, initializing [ 3615.115454] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3615.130762] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3619.744931] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 3631.864172] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3631.998693] Lustre: 119431:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3632.092886] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3632.103983] Lustre: Skipped 1 previous similar message [ 3632.264181] Lustre: lustre-MDT0001: new disk, initializing [ 3632.413254] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3632.456470] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3632.476034] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3637.492533] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 3648.234118] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3654.518875] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3662.239492] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3662.632557] Lustre: lustre-OST0000: new disk, initializing [ 3662.646661] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3662.662476] Lustre: 121400:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3662.799458] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3670.124928] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3670.138041] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3670.244751] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3670.507548] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 3680.840628] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3685.650488] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3685.862700] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3690.754682] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3691.029399] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3696.096660] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3696.098816] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3696.114624] LustreError: Skipped 1 previous similar message [ 3696.115280] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3696.115284] Lustre: Skipped 2 previous similar messages [ 3696.181207] Lustre: Skipped 2 previous similar messages [ 3701.227935] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3701.240507] Lustre: Skipped 2 previous similar messages [ 3706.364175] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3706.370185] Lustre: Skipped 1 previous similar message [ 3707.359387] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3707.558421] Lustre: server umount lustre-OST0000 complete [ 3714.530950] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3714.546375] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3714.561982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3718.164702] Lustre: server umount lustre-MDT0000 complete [ 3721.700335] LustreError: 119438:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3721.738210] LustreError: 119438:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3722.286570] LustreError: 119422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440841 with bad export cookie 5854019485244817372 [ 3722.298371] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3722.304163] LustreError: 119422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 3722.668784] Lustre: server umount lustre-MDT0001 complete [ 3730.950847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3731.531885] LustreError: 123619:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3731.618717] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3735.618527] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 3737.068707] LustreError: 123620:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3741.157233] LustreError: 123619:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3744.832060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3749.817496] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 3756.724987] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3762.850130] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3769.496046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3769.827054] LustreError: 125445:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3776.933397] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 3785.643678] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3793.206619] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3801.057238] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3801.070595] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3801.087457] Lustre: Skipped 1 previous similar message [ 3801.091406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3801.096875] Lustre: Skipped 2 previous similar messages [ 3804.393578] Lustre: server umount lustre-OST0000 complete [ 3811.808643] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3811.825640] LustreError: Skipped 1 previous similar message [ 3811.832376] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3811.845602] Lustre: Skipped 1 previous similar message [ 3816.859823] Lustre: server umount lustre-MDT0000 complete [ 3817.440681] LustreError: 123620:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3817.475675] LustreError: 123620:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3820.599518] LustreError: 123600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440939 with bad export cookie 5854019485244818261 [ 3820.603738] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3820.607051] LustreError: 123600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3820.934963] Lustre: server umount lustre-MDT0001 complete [ 3832.406437] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 3835.079658] Key type lgssc unregistered [ 3835.366391] LNet: 127254:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3835.388582] LNetError: 127254:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3836.454740] LNet: Removed LNI 192.168.206.117@tcp [ 3837.398283] Key type .llcrypt unregistered [ 3837.401252] Key type ._llcrypt unregistered [ 3856.423047] Key type ._llcrypt registered [ 3856.426356] Key type .llcrypt registered [ 3856.527172] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [ 3873.562168] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 3874.191046] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3874.269746] alg: No test for adler32 (adler32-zlib) [ 3875.561391] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3875.829442] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 3877.503358] Key type lgssc registered [ 3878.889801] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3923.827790] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3923.852883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3925.386364] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3925.429256] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3925.533687] Lustre: lustre-MDT0000: new disk, initializing [ 3925.610135] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3925.634173] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3930.488918] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 3942.057750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3942.143587] Lustre: 130408:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3942.177873] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3942.180769] Lustre: Skipped 1 previous similar message [ 3942.266053] Lustre: lustre-MDT0001: new disk, initializing [ 3942.354930] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3942.397876] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3942.410547] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3946.524686] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 3957.374743] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3964.458384] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3972.425246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3972.785936] Lustre: lustre-OST0000: new disk, initializing [ 3972.792611] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3972.799343] Lustre: 132376:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3972.871664] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3979.103923] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 3979.342417] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3979.356051] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3979.505381] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3988.539842] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3993.341226] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3993.595339] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3998.384540] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3998.620348] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4005.344539] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4005.352556] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4005.364197] LustreError: Skipped 1 previous similar message [ 4005.366399] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4005.392320] Lustre: Skipped 1 previous similar message [ 4006.571769] Lustre: server umount lustre-OST0000 complete [ 4014.048436] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4014.055686] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4014.076607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4014.086594] Lustre: Skipped 1 previous similar message [ 4015.584089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4015.588556] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4017.087988] Lustre: server umount lustre-MDT0000 complete [ 4020.982137] LustreError: 130400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441139 with bad export cookie 13317605229812580397 [ 4020.989410] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4020.994166] LustreError: 130400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4021.298620] Lustre: server umount lustre-MDT0001 complete [ 4030.551887] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 09:12:28 (1788441148) [ 4038.313260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4038.715183] LustreError: 134789:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4038.737987] LustreError: 134789:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4038.795304] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4042.774687] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 4044.259021] LustreError: 134790:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4048.361224] LustreError: 134789:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4051.595692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4052.036055] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4057.309278] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 4065.235568] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4071.649938] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4077.025841] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4077.061090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4077.071769] Lustre: Skipped 2 previous similar messages [ 4082.145883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4082.154087] Lustre: Skipped 1 previous similar message [ 4088.288355] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4088.384739] Lustre: server umount lustre-MDT0000 complete [ 4091.963434] LustreError: 134769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441210 with bad export cookie 13317605229812581272 [ 4091.974866] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4091.980894] LustreError: 134769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4092.444902] Lustre: server umount lustre-MDT0001 complete [ 4100.280208] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 09:13:38 (1788441218) [ 4106.667781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4122.208241] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4127.328349] LustreError: 137207:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.117@tcp: failed processing log, type 4: rc = -110 [ 4153.056022] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4158.514848] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 4165.208440] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4167.346083] Lustre: server umount lustre-OST0000 complete [ 4176.143494] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 09:14:54 (1788441294) [ 4182.772098] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4183.343217] LustreError: 138652:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4183.418617] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4187.594852] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 4188.643125] LustreError: 138653:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4196.313754] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4201.248658] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 4209.194562] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4215.165236] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4221.526272] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4221.857422] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4221.869886] Lustre: Skipped 1 previous similar message [ 4227.608377] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 4234.232739] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4253.156030] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4253.171718] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4253.189117] Lustre: Skipped 1 previous similar message [ 4253.211555] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4253.219747] Lustre: Skipped 2 previous similar messages [ 4258.335708] Lustre: server umount lustre-MDT0000 complete [ 4261.861745] LustreError: 138652:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4261.881944] LustreError: 138652:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4262.262776] LustreError: 138632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441381 with bad export cookie 13317605229812581944 [ 4262.269047] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4262.271233] LustreError: 138632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4262.607506] Lustre: server umount lustre-MDT0001 complete [ 4276.841228] Lustre: server umount lustre-OST0000 complete [ 4286.699280] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 09:16:44 (1788441404) [ 4293.829932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4294.327800] LustreError: 142329:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4294.390644] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4299.430384] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 4308.552398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4313.523860] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 4321.639929] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4327.433208] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4334.007284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4339.444145] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4339.939701] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 4346.519114] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4351.359187] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4351.555968] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4355.725622] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4356.025140] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4359.136786] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4359.139975] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4359.161825] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4359.171725] Lustre: Skipped 3 previous similar messages [ 4359.190811] Lustre: Skipped 4 previous similar messages [ 4364.030354] Lustre: server umount lustre-OST0000 complete [ 4374.500204] LustreError: 142330:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4374.509800] LustreError: 142330:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4374.550699] Lustre: server umount lustre-MDT0000 complete [ 4378.488593] LustreError: 145641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441497 with bad export cookie 13317605229812583015 [ 4378.493110] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4378.494771] LustreError: 145641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4378.915151] Lustre: server umount lustre-MDT0001 complete [ 4389.439575] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 09:18:26 (1788441506) [ 4397.074335] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4412.641303] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4417.761537] LustreError: 146488:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.117@tcp: failed processing log, type 4: rc = -110 [ 4443.487267] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4443.495961] Lustre: Skipped 2 previous similar messages [ 4449.526708] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 4456.560161] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4463.162712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4463.633932] LustreError: 146514:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4463.804453] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4468.620594] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 4476.956741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4481.829215] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 4490.998854] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4497.340626] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4502.063723] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4502.262956] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4507.488897] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4507.743711] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4513.248374] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4513.266389] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4513.274916] Lustre: Skipped 2 previous similar messages [ 4513.286078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4513.290555] Lustre: Skipped 2 previous similar messages [ 4516.502542] Lustre: server umount lustre-MDT0000 complete [ 4520.525741] LustreError: 146496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441639 with bad export cookie 13317605229812584037 [ 4520.527109] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4520.547413] LustreError: 146496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4520.942774] Lustre: server umount lustre-MDT0001 complete [ 4535.286234] Lustre: server umount lustre-OST0000 complete [ 4544.548132] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 09:21:02 (1788441662) [ 4551.886794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4567.583727] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4572.703562] LustreError: 150683:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.117@tcp: failed processing log, type 4: rc = -110 [ 4604.996721] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 4613.257709] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4620.465137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4620.963770] LustreError: 150708:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4620.979048] LustreError: 150708:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 4621.114874] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4626.151653] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 4635.107582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4641.082134] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 4649.940371] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4655.795042] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4662.567795] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4662.732197] Lustre: lustre-OST0001: new disk, initializing [ 4662.735380] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4662.740104] Lustre: 153558:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4669.378791] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 4669.528476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4669.547453] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4669.709818] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4678.403715] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4682.792678] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4682.948061] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4687.266487] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4687.475313] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4690.409856] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4690.417697] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4690.423037] Lustre: Skipped 3 previous similar messages [ 4690.438455] Lustre: Skipped 4 previous similar messages [ 4695.341839] Lustre: server umount lustre-OST0000 complete [ 4700.641894] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4700.652572] LustreError: Skipped 1 previous similar message [ 4705.162725] Lustre: server umount lustre-OST0001 complete [ 4714.613877] Lustre: server umount lustre-MDT0000 complete [ 4717.544825] LustreError: 150689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441836 with bad export cookie 13317605229812584989 [ 4717.556761] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4717.559815] LustreError: 150689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4717.847283] Lustre: server umount lustre-MDT0001 complete [ 4740.994685] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 09:24:18 (1788441858) [ 4742.765386] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4744.419637] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4746.107134] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 09:24:23 (1788441863) [ 4752.992528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4753.362539] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4753.404227] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4753.656553] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4753.666546] Lustre: Skipped 6 previous similar messages [ 4758.492051] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 4768.012710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4768.143785] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4768.174609] Lustre: 157116:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4773.246225] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 4780.633445] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4786.374266] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4793.289894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4793.525272] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4799.174693] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4799.742685] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 4807.873592] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4812.336375] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4812.528366] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4816.795631] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4816.956833] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4819.937066] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4819.942305] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4819.953939] LustreError: Skipped 1 previous similar message [ 4819.956824] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4819.975758] Lustre: Skipped 5 previous similar messages [ 4819.995575] Lustre: Skipped 5 previous similar messages [ 4833.247213] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4833.437524] Lustre: server umount lustre-OST0000 complete [ 4855.940699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4860.328106] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4860.346863] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4860.349166] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4860.377056] Lustre: Skipped 1 previous similar message [ 4862.785947] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 4870.334724] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4875.293740] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4875.608444] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4879.985604] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4880.197278] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4885.269385] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9a698923d800.ost_server_uuid 50 [ 4886.424948] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9a698923d800.ost_server_uuid in FULL state after 0 sec [ 4886.677292] LustreError: 157121:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.206.17@tcp) returned error from blocking AST (req@00000000a54e7877 x1875316303990400 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000231eab50/0xb8d1a36595dd1b77 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.206.17@tcp remote: 0xc13d4d78e37577f0 expref: 5 pid: 158983 timeout: 4986 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4886.727305] LustreError: lustre-MDT0000: A client on nid 192.168.206.17@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4886.736265] LustreError: 157111:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.206.17@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000231eab50/0xb8d1a36595dd1b77 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.206.17@tcp remote: 0xc13d4d78e37577f0 expref: 6 pid: 158983 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4891.104429] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4891.120507] LustreError: Skipped 1 previous similar message [ 4896.785810] Lustre: server umount lustre-OST0000 complete [ 4906.565167] Lustre: server umount lustre-MDT0000 complete [ 4910.064488] LustreError: 157856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442028 with bad export cookie 13317605229812586172 [ 4910.071817] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4910.078952] LustreError: 157856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4924.383441] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4924.636746] Lustre: server umount lustre-MDT0001 complete [ 4933.834587] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 4936.741792] Key type lgssc unregistered [ 4936.987605] LNet: 163229:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4937.001991] LNetError: 163229:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4937.025809] LNet: Removed LNI 192.168.206.117@tcp [ 4937.886172] Key type .llcrypt unregistered [ 4937.889180] Key type ._llcrypt unregistered [ 4960.179922] Key type ._llcrypt registered [ 4960.182498] Key type .llcrypt registered [ 4960.335189] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 4961.187469] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4961.202275] alg: No test for adler32 (adler32-zlib) [ 4962.333312] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 4962.581500] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 4964.263195] Key type lgssc registered [ 4965.419967] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4972.760762] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4972.773869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4974.218786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4978.460994] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 4984.803066] LustreError: 164639:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4984.828430] LustreError: 164639:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4987.399189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4987.774050] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4992.221898] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 4999.527757] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5005.562980] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5012.239985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5012.574571] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5018.933855] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 5019.756489] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5025.804392] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5032.247713] Lustre: Failing over lustre-MDT0000 [ 5032.681628] Lustre: server umount lustre-MDT0000 complete [ 5033.497184] LustreError: 164633:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5033.971667] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5033.982934] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5035.489587] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5035.490843] LustreError: 164639:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5035.507824] Lustre: Skipped 1 previous similar message [ 5035.517931] LustreError: 164639:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5040.626473] LustreError: 164634:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5040.691752] LustreError: 164634:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5045.388301] LustreError: 164620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.206.17@tcp arrived at 1788442164 with bad export cookie 17712741919163911394 [ 5045.397489] LustreError: 164620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5045.729644] LustreError: 164639:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5045.746094] LustreError: 164639:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5051.871192] Lustre: 163938:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788442154/real 1788442154] req@0000000087d251f0 x1875317443552640/t0(0) o400->MGC192.168.206.117@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788442170 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5051.893549] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5053.227372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5057.003283] LustreError: 164638:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5057.022312] LustreError: 164638:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 5062.116993] LustreError: 163937:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000005935f214 x1875317443558528/t0(0) o250->MGC192.168.206.117@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5062.382137] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5062.445792] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5066.842706] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 5067.747233] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5067.762897] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5068.861983] Lustre: lustre-MDT0000: Denying connection for new client d1ccd7e5-4806-4511-829c-541ecfcf0cdd (at 192.168.206.17@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 5073.916845] Lustre: lustre-MDT0000: Denying connection for new client d1ccd7e5-4806-4511-829c-541ecfcf0cdd (at 192.168.206.17@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:03 [ 5077.989318] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5077.994810] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5078.004370] Lustre: Skipped 1 previous similar message [ 5078.009617] Lustre: Skipped 1 previous similar message [ 5082.833758] Lustre: server umount lustre-OST0000 complete [ 5086.200804] LustreError: 168373:0:(ldlm_lib.c:3029:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5086.206549] Lustre: 167720:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5086.211785] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5086.225429] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5086.236701] LustreError: 167720:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000f0b05f5c x1875317443572864/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5086.251497] LustreError: 167720:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5086.259845] LustreError: 167720:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5086.267643] Lustre: lustre-MDT0000: Recovery over after 0:19, of 2 clients 0 recovered and 2 were evicted. [ 5086.473034] Lustre: server umount lustre-MDT0000 complete [ 5088.224449] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5088.243908] LustreError: 167133:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5088.265740] LustreError: 167133:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 5090.234688] LustreError: 167339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442209 with bad export cookie 17712741919163911492 [ 5090.240829] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5090.245148] LustreError: 167339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5090.257183] LustreError: 163937:0:(import.c:751:ptlrpc_connect_import_locked()) can't connect to a closed import [ 5092.593908] Lustre: server umount lustre-MDT0001 complete [ 5101.753651] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 5104.107455] Key type lgssc unregistered [ 5104.368258] LNet: 169123:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5104.381813] LNetError: 169123:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5105.445141] LNet: Removed LNI 192.168.206.117@tcp [ 5106.298595] Key type .llcrypt unregistered [ 5106.301574] Key type ._llcrypt unregistered [ 5126.880537] Key type ._llcrypt registered [ 5126.884976] Key type .llcrypt registered [ 5126.980840] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 5127.874307] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5128.029854] alg: No test for adler32 (adler32-zlib) [ 5129.168320] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 5129.361313] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 5131.031402] Key type lgssc registered [ 5132.198738] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5139.627252] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5139.651054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5141.104358] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5145.125579] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 5150.696780] LustreError: 170510:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5150.708166] LustreError: 170510:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5152.654731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5153.083614] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5157.080428] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 5164.004163] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5168.686209] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5174.835438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5175.129818] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5180.800972] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 5185.397950] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5187.628396] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5195.760210] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5195.781720] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5195.788604] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5195.804150] Lustre: Skipped 1 previous similar message [ 5199.632623] Lustre: server umount lustre-OST0000 complete [ 5204.459932] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5204.469535] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5204.499776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5204.510584] Lustre: Skipped 1 previous similar message [ 5205.985155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5205.986769] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5209.710981] Lustre: server umount lustre-MDT0000 complete [ 5211.108201] LustreError: 170509:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5211.125917] LustreError: 170509:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5213.411318] LustreError: 170491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442332 with bad export cookie 4998753821695235918 [ 5213.425620] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5213.793747] Lustre: server umount lustre-MDT0001 complete [ 5223.578572] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 5225.982164] Key type lgssc unregistered [ 5226.263201] LNet: 174039:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5226.276927] LNetError: 174039:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5227.304380] LNet: Removed LNI 192.168.206.117@tcp [ 5228.047385] Key type .llcrypt unregistered [ 5228.049602] Key type ._llcrypt unregistered [ 5261.092162] Key type ._llcrypt registered [ 5261.094495] Key type .llcrypt registered [ 5261.213769] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 5262.437865] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5262.500486] alg: No test for adler32 (adler32-zlib) [ 5263.695782] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 5263.933462] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 5265.591164] Key type lgssc registered [ 5266.815795] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5274.410858] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5274.442432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5275.882137] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5280.318645] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 5285.349973] LustreError: 175736:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5285.372231] LustreError: 175736:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5288.297497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5288.817237] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5293.104354] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 5300.192416] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5305.489333] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5311.944079] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5312.317967] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5318.772429] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 5320.501750] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5327.479846] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5336.037702] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5336.040261] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5336.056770] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5337.342310] Lustre: server umount lustre-OST0000 complete [ 5345.248347] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5345.258804] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5345.275353] Lustre: Skipped 1 previous similar message [ 5345.281124] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5345.286805] Lustre: Skipped 1 previous similar message [ 5346.275341] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5346.292116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5346.298077] Lustre: Skipped 1 previous similar message [ 5347.597012] Lustre: server umount lustre-MDT0000 complete [ 5351.136654] LustreError: 177580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442470 with bad export cookie 5138172257930371595 [ 5351.143094] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5351.144876] LustreError: 177580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5351.475877] Lustre: server umount lustre-MDT0001 complete [ 5362.502809] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 5365.198776] Key type lgssc unregistered [ 5365.466410] LNet: 179269:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5365.481090] LNetError: 179269:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5365.493967] LNet: Removed LNI 192.168.206.117@tcp [ 5366.411578] Key type .llcrypt unregistered [ 5366.413777] Key type ._llcrypt unregistered [ 5387.196369] Key type ._llcrypt registered [ 5387.206399] Key type .llcrypt registered [ 5387.335245] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 5388.233031] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5388.266636] alg: No test for adler32 (adler32-zlib) [ 5389.331369] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 5389.604372] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 5391.311233] Key type lgssc registered [ 5392.473601] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5401.914102] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5401.937786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5403.323574] Lustre: *** cfs_fail_loc=135, val=0*** [ 5403.390433] LustreError: 180781:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5403.398410] LustreError: 180781:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.117@tcp: cfg command failed: rc = -2 [ 5403.403531] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5403.403531] [ 5403.412412] LustreError: MGC192.168.206.117@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5403.429506] LustreError: 180742:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5403.435940] LustreError: 180742:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [ 5403.444977] LustreError: 180742:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 5403.452841] LustreError: 180742:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5403.514918] Lustre: server umount lustre-MDT0000 complete [ 5403.525559] LustreError: 180742:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5410.432087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5410.875541] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5415.927796] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 5421.033666] LustreError: 181152:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5421.055454] LustreError: 181152:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5425.224870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5425.610665] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5430.349555] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 5438.303537] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5444.140345] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5450.595382] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5450.600125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5450.607829] Lustre: Skipped 1 previous similar message [ 5452.815993] Lustre: server umount lustre-MDT0000 complete [ 5455.842291] LustreError: 182392:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5455.864017] LustreError: 182392:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5456.887776] LustreError: 181132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442575 with bad export cookie 17973779161121249385 [ 5456.908289] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5456.909425] LustreError: 181132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5457.273803] Lustre: server umount lustre-MDT0001 complete [ 5468.364853] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 5470.824188] Key type lgssc unregistered [ 5471.047115] LNet: 183471:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5471.053697] LNetError: 183471:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5471.069059] LNet: Removed LNI 192.168.206.117@tcp [ 5472.025464] Key type .llcrypt unregistered [ 5472.027962] Key type ._llcrypt unregistered [ 5492.923843] Key type ._llcrypt registered [ 5492.926666] Key type .llcrypt registered [ 5493.041664] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 5493.428442] Key type .llcrypt unregistered [ 5493.436991] Key type ._llcrypt unregistered [ 5510.127436] Key type ._llcrypt registered [ 5510.130297] Key type .llcrypt registered [ 5510.241986] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 5511.471220] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5511.558879] alg: No test for adler32 (adler32-zlib) [ 5512.609449] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 5512.936367] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 5514.647156] Key type lgssc registered [ 5515.871267] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5523.337112] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5523.355182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5538.915668] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5544.032199] LustreError: 185597:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.117@tcp: failed processing log, type 4: rc = -110 [ 5570.721019] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5576.721807] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 5585.602221] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5592.345741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5592.909918] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5593.012727] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5597.264548] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 5598.178492] LustreError: 185623:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5598.212095] LustreError: 185623:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5603.297903] LustreError: 185622:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5605.359078] LustreError: 185623:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5605.386952] LustreError: 185623:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5606.468831] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5606.839875] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5611.514301] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 5620.034952] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5626.933227] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5629.945080] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5642.729470] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5642.732109] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5642.782250] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5642.800733] Lustre: Skipped 1 previous similar message [ 5647.861681] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5647.873236] Lustre: Skipped 1 previous similar message [ 5648.819365] Lustre: server umount lustre-OST0000 complete [ 5652.960465] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5652.972281] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5652.974204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5652.994200] Lustre: Skipped 2 previous similar messages [ 5658.079985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5658.086460] Lustre: Skipped 2 previous similar messages [ 5663.203635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5663.210546] Lustre: Skipped 1 previous similar message [ 5667.295972] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5667.393418] Lustre: server umount lustre-MDT0000 complete [ 5668.321910] LustreError: 186639:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5668.349055] LustreError: 186639:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5671.240889] LustreError: 185605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442790 with bad export cookie 11584353216166152507 [ 5671.243573] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5671.252685] LustreError: 185605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5671.618964] Lustre: server umount lustre-MDT0001 complete [ 5683.569198] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 5686.400647] Key type lgssc unregistered [ 5686.693078] LNet: 189612:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5686.706144] LNetError: 189612:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5686.732946] LNet: Removed LNI 192.168.206.117@tcp [ 5687.683473] Key type .llcrypt unregistered [ 5687.692038] Key type ._llcrypt unregistered [ 5712.412788] Key type ._llcrypt registered [ 5712.417515] Key type .llcrypt registered [ 5712.514801] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 5713.911431] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5713.973606] alg: No test for adler32 (adler32-zlib) [ 5715.012920] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 5715.289045] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 5716.983163] Key type lgssc registered [ 5718.362504] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5727.097124] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5727.116110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5728.710101] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5733.451758] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 5738.992133] LustreError: 191020:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5739.020641] LustreError: 191020:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5742.222862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5742.560807] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5747.153057] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 5755.654960] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5762.789039] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5770.421644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5770.758178] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5773.836358] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 5778.247993] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 5786.415929] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5791.975854] Lustre: Failing over lustre-MDT0000 [ 5792.421185] Lustre: server umount lustre-MDT0000 complete [ 5793.248716] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5793.251253] LustreError: 192250:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5793.272299] Lustre: Skipped 1 previous similar message [ 5793.298326] LustreError: 192250:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5794.317223] LustreError: 191015:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5798.369049] LustreError: 191017:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5798.394819] LustreError: 191017:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5803.491674] LustreError: 191016:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5803.512446] LustreError: 191016:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5809.631151] Lustre: 190333:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788442912/real 1788442912] req@0000000070a7c350 x1875318233130624/t0(0) o400->MGC192.168.206.117@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788442928 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5809.682488] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5810.969069] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5814.697815] LustreError: 192250:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5814.731218] LustreError: 192250:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 5819.890802] LustreError: 190329:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000997a3ff4 x1875318233136128/t0(0) o250->MGC192.168.206.117@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5819.929761] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5819.956537] Lustre: MGS: Client 0a5aa8e2-6543-478a-bacc-989975ac9605 (at 0@lo) reconnecting [ 5819.977437] Lustre: MGC192.168.206.117@tcp: Connection restored to 0@lo (at 0@lo) [ 5820.241558] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5820.312493] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5824.914609] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 5825.006316] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5825.520837] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5825.532223] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 5825.578295] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 5829.803625] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 5835.695826] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 5845.989559] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5845.994878] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5846.001358] Lustre: Skipped 2 previous similar messages [ 5846.018651] Lustre: Skipped 1 previous similar message [ 5846.934252] Lustre: server umount lustre-OST0000 complete [ 5851.104873] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5851.106927] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5851.115565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5851.127089] Lustre: Skipped 1 previous similar message [ 5851.135744] Lustre: Skipped 1 previous similar message [ 5856.224765] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5856.236672] Lustre: Skipped 1 previous similar message [ 5857.168981] Lustre: server umount lustre-MDT0000 complete [ 5860.861959] LustreError: 191000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442979 with bad export cookie 6150949207441402695 [ 5860.868755] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5860.875562] LustreError: 191000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5861.227467] Lustre: server umount lustre-MDT0001 complete [ 5874.718225] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 5877.649155] Key type lgssc unregistered [ 5877.976421] LNet: 195862:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5877.983634] LNetError: 195862:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5878.006840] LNet: Removed LNI 192.168.206.117@tcp [ 5879.056475] Key type .llcrypt unregistered [ 5879.060771] Key type ._llcrypt unregistered [ 5902.811877] Key type ._llcrypt registered [ 5902.814313] Key type .llcrypt registered [ 5902.915785] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 5903.996603] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5904.049307] alg: No test for adler32 (adler32-zlib) [ 5905.182428] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 5905.441292] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 5907.087155] Key type lgssc registered [ 5908.218126] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5916.377909] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5916.394226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5917.783933] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5922.468620] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 5927.910411] LustreError: 197270:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5927.932188] LustreError: 197270:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5930.703590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5931.099649] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5935.294711] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 5943.437204] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5949.713607] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5957.239601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5957.531380] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5962.680899] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 5964.705971] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 5973.214909] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5976.762882] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 5985.767681] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6000.667243] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6006.757897] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6006.758801] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6006.787334] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6006.793868] Lustre: Skipped 1 previous similar message [ 6011.879217] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6011.892691] Lustre: Skipped 1 previous similar message [ 6012.996253] Lustre: server umount lustre-OST0000 complete [ 6017.007735] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6017.025440] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6017.052249] Lustre: Skipped 2 previous similar messages [ 6017.080069] Lustre: Skipped 1 previous similar message [ 6020.065981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6020.071305] Lustre: Skipped 1 previous similar message [ 6022.741834] Lustre: server umount lustre-MDT0000 complete [ 6026.208217] LustreError: 197270:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6026.228728] LustreError: 197270:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6026.298157] LustreError: 197249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443145 with bad export cookie 7277963192171620614 [ 6026.312284] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6026.323551] LustreError: 197249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6026.558850] Lustre: server umount lustre-MDT0001 complete [ 6037.448128] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 6040.308600] Key type lgssc unregistered [ 6040.678850] LNet: 201119:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6040.686261] LNetError: 201119:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6040.702137] LNet: Removed LNI 192.168.206.117@tcp [ 6041.594950] Key type .llcrypt unregistered [ 6041.597835] Key type ._llcrypt unregistered [ 6061.713421] Key type ._llcrypt registered [ 6061.715917] Key type .llcrypt registered [ 6061.834847] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 6062.788091] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6062.866921] alg: No test for adler32 (adler32-zlib) [ 6063.961713] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 6064.248470] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 6065.911194] Key type lgssc registered [ 6066.996789] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6075.079633] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6075.097601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6076.557350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6080.981623] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6086.116897] LustreError: 202379:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6086.143231] LustreError: 202379:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6089.236475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6089.549082] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6094.050620] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6101.268188] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6106.894780] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6113.233399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6113.624859] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6119.160505] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6121.852696] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6125.893873] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6131.841199] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6142.596508] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6152.131683] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6159.650275] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6171.107386] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6171.113699] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6171.129047] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6173.162953] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6173.167871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6173.178549] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6173.203735] Lustre: Skipped 1 previous similar message [ 6175.743457] Lustre: server umount lustre-OST0000 complete [ 6181.857213] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6181.862360] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6181.883284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6185.578744] Lustre: server umount lustre-MDT0000 complete [ 6188.513101] LustreError: 202374:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6188.526080] LustreError: 202374:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6189.088567] LustreError: 202359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443307 with bad export cookie 7309731462370279248 [ 6189.099677] LustreError: 202359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6189.105687] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6189.376353] Lustre: server umount lustre-MDT0001 complete [ 6199.536518] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 6202.322354] Key type lgssc unregistered [ 6202.646214] LNet: 207402:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6202.658848] LNetError: 207402:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6202.677913] LNet: Removed LNI 192.168.206.117@tcp [ 6203.656446] Key type .llcrypt unregistered [ 6203.661476] Key type ._llcrypt unregistered [ 6228.765629] Key type ._llcrypt registered [ 6228.769373] Key type .llcrypt registered [ 6228.886178] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 6230.559847] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6230.633427] alg: No test for adler32 (adler32-zlib) [ 6231.717397] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 6232.005798] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 6233.727227] Key type lgssc registered [ 6234.872205] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6242.564113] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6242.584146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6244.142886] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6248.190200] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6253.547647] LustreError: 208819:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6253.566326] LustreError: 208819:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6256.959598] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6257.331311] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6261.713734] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6268.953657] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6275.062297] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6281.484030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6282.007369] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6287.703586] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6288.188161] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6294.903836] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6308.832912] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6308.838183] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6308.844451] LustreError: Skipped 1 previous similar message [ 6308.845324] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6308.856806] Lustre: Skipped 1 previous similar message [ 6308.877106] Lustre: Skipped 1 previous similar message [ 6313.962110] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6313.977100] Lustre: Skipped 1 previous similar message [ 6314.351548] Lustre: server umount lustre-OST0000 complete [ 6319.072428] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6319.076220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6319.090515] Lustre: Skipped 1 previous similar message [ 6319.101053] Lustre: Skipped 1 previous similar message [ 6324.193797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6324.199989] Lustre: Skipped 1 previous similar message [ 6329.313371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6329.319263] Lustre: Skipped 1 previous similar message [ 6332.383188] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6332.553454] Lustre: server umount lustre-MDT0000 complete [ 6334.438913] LustreError: 209871:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6334.468193] LustreError: 209871:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6336.144775] LustreError: 208799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443455 with bad export cookie 14430037421157702374 [ 6336.149940] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6336.161289] LustreError: 208799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6336.520951] Lustre: server umount lustre-MDT0001 complete [ 6347.519835] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 6350.328626] Key type lgssc unregistered [ 6350.625193] LNet: 212502:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6350.638310] LNetError: 212502:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6350.669295] LNet: Removed LNI 192.168.206.117@tcp [ 6351.591173] Key type .llcrypt unregistered [ 6351.592848] Key type ._llcrypt unregistered [ 6369.253777] Key type ._llcrypt registered [ 6369.259402] Key type .llcrypt registered [ 6369.381918] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 6370.154392] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6370.170215] alg: No test for adler32 (adler32-zlib) [ 6371.232737] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 6371.427712] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 6373.072239] Key type lgssc registered [ 6373.989597] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6380.996337] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6381.014236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6382.336054] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6386.656575] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6392.807700] LustreError: 213767:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6392.824300] LustreError: 213767:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6394.555908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6394.982950] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6399.636758] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6406.986465] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6412.163331] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6418.904694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6419.219071] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6426.193395] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6427.452225] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6434.562587] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6451.169321] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6451.176845] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6451.196022] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6453.218962] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6453.219483] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6453.220313] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6456.922807] Lustre: server umount lustre-OST0000 complete [ 6461.410141] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6461.426162] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6461.441887] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6461.450542] Lustre: Skipped 1 previous similar message [ 6463.458044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6463.460264] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6467.614102] Lustre: server umount lustre-MDT0000 complete [ 6468.581714] LustreError: 213762:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6468.599749] LustreError: 213762:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6471.505716] LustreError: 213746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443590 with bad export cookie 8685499675166794618 [ 6471.508131] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6471.519525] LustreError: 213746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6471.909533] Lustre: server umount lustre-MDT0001 complete [ 6483.177537] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 6486.379414] Key type lgssc unregistered [ 6486.702022] LNet: 217450:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6486.708401] LNetError: 217450:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6486.733438] LNet: Removed LNI 192.168.206.117@tcp [ 6487.611684] Key type .llcrypt unregistered [ 6487.615505] Key type ._llcrypt unregistered [ 6507.580295] Key type ._llcrypt registered [ 6507.592458] Key type .llcrypt registered [ 6507.900339] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 6508.948738] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6509.432504] alg: No test for adler32 (adler32-zlib) [ 6510.647121] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 6511.029047] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 6512.800247] Key type lgssc registered [ 6514.170843] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6521.985256] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6522.018767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6523.547341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6528.354463] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6534.118853] LustreError: 218712:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6534.156886] LustreError: 218712:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6536.891339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6537.174593] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6541.552325] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6547.812305] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6554.552725] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6563.729506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6564.411135] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6569.543166] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6571.726215] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6579.786081] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6588.813212] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6588.885662] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6589.123574] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6595.693987] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6603.587337] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6605.341578] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6617.221182] Lustre: Permanently deactivating lustre-OST0001 [ 6617.248824] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6623.305464] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6629.050290] Lustre: Permanently reactivating lustre-OST0001 [ 6629.075060] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6629.080901] Lustre: Skipped 2 previous similar messages [ 6630.110499] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6630.136584] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6630.142192] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6630.152199] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6630.219178] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6640.608926] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6640.611130] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6640.628775] LustreError: Skipped 1 previous similar message [ 6640.630825] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6640.651177] Lustre: Skipped 2 previous similar messages [ 6642.656500] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6642.673320] Lustre: Skipped 2 previous similar messages [ 6646.198401] Lustre: server umount lustre-OST0001 complete [ 6647.788579] LustreError: 220535:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6647.807340] LustreError: 220535:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6651.363846] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6651.371797] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6651.379535] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6652.904287] LustreError: 221251:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6652.941768] LustreError: 221251:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6655.999944] Lustre: server umount lustre-OST0000 complete [ 6660.064372] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6660.077401] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6660.089610] Lustre: Skipped 1 previous similar message [ 6660.099611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6660.107016] Lustre: Skipped 2 previous similar messages [ 6668.256879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6668.268858] Lustre: Skipped 3 previous similar messages [ 6674.399129] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6674.688506] Lustre: server umount lustre-MDT0000 complete [ 6677.473946] LustreError: 218712:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6677.489137] LustreError: 218712:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6678.261281] LustreError: 218694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443797 with bad export cookie 16194480832691580726 [ 6678.263609] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6678.280414] LustreError: 218694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6678.621811] Lustre: server umount lustre-MDT0001 complete [ 6688.580571] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 6691.645682] Key type lgssc unregistered [ 6692.017656] LNet: 223759:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6692.029807] LNetError: 223759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6692.042776] LNet: Removed LNI 192.168.206.117@tcp [ 6692.957315] Key type .llcrypt unregistered [ 6692.962915] Key type ._llcrypt unregistered [ 6727.380514] Key type ._llcrypt registered [ 6727.382936] Key type .llcrypt registered [ 6727.467771] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 6728.293284] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6728.309596] alg: No test for adler32 (adler32-zlib) [ 6729.280531] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 6729.455893] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 6731.095609] Key type lgssc registered [ 6732.084551] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6739.359710] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6739.373617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6740.581282] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6740.600137] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6740.763834] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6745.541847] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6754.097719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6754.218080] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6754.267285] Lustre: 225940:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6754.577602] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6759.534073] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6767.601307] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6773.925943] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6781.034712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6781.203392] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6781.391627] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6786.984538] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6791.343720] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 6793.696836] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6796.610894] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6804.270523] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6809.867474] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6816.539525] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6829.836621] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6829.844283] Lustre: Skipped 1 previous similar message [ 6851.910909] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6851.921239] Lustre: Skipped 2 previous similar messages [ 6891.033758] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6891.039759] Lustre: Skipped 4 previous similar messages [ 6965.733740] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6965.734558] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6965.745262] LustreError: Skipped 1 previous similar message [ 6965.746215] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6965.757831] Lustre: Skipped 1 previous similar message [ 6967.681046] Lustre: server umount lustre-OST0000 complete [ 6974.945307] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6974.962373] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6974.993296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6974.995812] Lustre: Skipped 1 previous similar message [ 6975.975528] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6975.997541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6976.004013] Lustre: Skipped 1 previous similar message [ 6977.481920] Lustre: server umount lustre-MDT0000 complete [ 6981.352623] LustreError: 225930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444100 with bad export cookie 8864990303290748571 [ 6981.370654] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6981.379448] LustreError: 225930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6981.775065] Lustre: server umount lustre-MDT0001 complete [ 6997.512879] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 7000.126369] Key type lgssc unregistered [ 7000.403950] LNet: 230735:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7000.417888] LNetError: 230735:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7000.430318] LNet: Removed LNI 192.168.206.117@tcp [ 7001.384076] Key type .llcrypt unregistered [ 7001.388349] Key type ._llcrypt unregistered [ 7024.401540] Key type ._llcrypt registered [ 7024.403304] Key type .llcrypt registered [ 7024.481041] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 7025.551144] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7025.573044] alg: No test for adler32 (adler32-zlib) [ 7026.595858] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 7026.822850] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 7028.495152] Key type lgssc registered [ 7029.468855] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7036.651122] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7036.687438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7038.076911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7042.670342] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 7047.660346] LustreError: 232147:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7047.695293] LustreError: 232147:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7051.213702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7051.512381] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7057.269582] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 7063.487357] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7069.022439] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7075.786501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7076.187662] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7082.366457] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7082.426205] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 7090.350831] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7113.185548] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7113.190367] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7113.214718] LustreError: Skipped 1 previous similar message [ 7113.219063] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7113.230157] Lustre: Skipped 1 previous similar message [ 7113.255355] Lustre: Skipped 1 previous similar message [ 7116.054468] Lustre: server umount lustre-OST0000 complete [ 7123.425076] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7123.479806] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7123.495083] Lustre: Skipped 1 previous similar message [ 7126.328325] Lustre: server umount lustre-MDT0000 complete [ 7128.546247] LustreError: 232148:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7128.566695] LustreError: 232148:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7130.713636] LustreError: 234799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444249 with bad export cookie 3731553793051120146 [ 7130.716559] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7130.723314] LustreError: 234799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7131.092486] Lustre: server umount lustre-MDT0001 complete [ 7141.094931] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 7143.233841] Key type lgssc unregistered [ 7143.482735] LNet: 235837:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7143.512072] LNetError: 235837:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7143.540574] LNet: Removed LNI 192.168.206.117@tcp [ 7144.270960] Key type .llcrypt unregistered [ 7144.273143] Key type ._llcrypt unregistered [ 7170.904510] Key type ._llcrypt registered [ 7170.911694] Key type .llcrypt registered [ 7171.083389] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 7172.165724] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7172.200062] alg: No test for adler32 (adler32-zlib) [ 7173.345767] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 7173.577616] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 7175.351146] Key type lgssc registered [ 7176.519376] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7446.981749] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 7449.211317] Key type lgssc unregistered [ 7449.457366] LNet: 237858:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7449.462189] LNetError: 237858:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7449.480891] LNet: Removed LNI 192.168.206.117@tcp [ 7450.437945] Key type .llcrypt unregistered [ 7450.439766] Key type ._llcrypt unregistered [ 7481.365559] Key type ._llcrypt registered [ 7481.367253] Key type .llcrypt registered [ 7481.433932] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [ 7498.144804] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 7499.158910] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7499.196413] alg: No test for adler32 (adler32-zlib) [ 7500.526620] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 7500.859620] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 7502.671907] Key type lgssc registered [ 7504.134330] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7535.279411] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7302 sec ========== 10:10:53 (1788444653) [ 7537.125750] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 10:10:54 (1788444654) === [ 7540.694290] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 10:10:58 (1788444658) === [ 7579.123174] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 7581.973647] Key type lgssc unregistered [ 7582.240229] LNet: 243091:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7582.251415] LNetError: 243091:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7582.277760] LNet: Removed LNI 192.168.206.117@tcp [ 7583.064932] Key type .llcrypt unregistered [ 7583.066696] Key type ._llcrypt unregistered