[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 527697614 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a2000-0x1465ccfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002271] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004010] kvm-guest: setup PV IPIs [ 0.006584] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008011] pid_max: default: 32768 minimum: 301 [ 0.009128] LSM: Security Framework initializing [ 0.010050] Yama: becoming mindful. [ 0.011024] SELinux: Initializing. [ 0.012044] *** VALIDATE selinux *** [ 0.019442] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024177] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025150] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026111] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027124] *** VALIDATE tmpfs *** [ 0.029351] *** VALIDATE proc *** [ 0.030228] *** VALIDATE cgroup *** [ 0.032006] *** VALIDATE cgroup2 *** [ 0.033267] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034173] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036030] Spectre V2 : User space: Vulnerable [ 0.037007] Speculative Store Bypass: Vulnerable [ 0.039758] debug: unmapping init [mem 0xffffffff96859000-0xffffffff96860fff] [ 0.041178] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042696] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043025] ... version: 2 [ 0.044028] ... bit width: 48 [ 0.045011] ... generic registers: 4 [ 0.046012] ... value mask: 0000ffffffffffff [ 0.047012] ... max period: 00007fffffffffff [ 0.048012] ... fixed-purpose events: 3 [ 0.049011] ... event mask: 000000070000000f [ 0.050314] rcu: Hierarchical SRCU implementation. [ 0.052436] smp: Bringing up secondary CPUs ... [ 0.053552] x86: Booting SMP configuration: [ 0.054022] .... node #0, CPUs: #1 #2 #3 [ 0.057688] smp: Brought up 1 node, 4 CPUs [ 0.059000] smpboot: Max logical packages: 1 [ 0.060011] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.138367] node 0 deferred pages initialised in 74ms [ 0.141098] devtmpfs: initialized [ 0.141909] x86/mm: Memory block size: 128MB [ 0.144198] gcov: version magic: 0x41383552 [ 0.145569] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.148085] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.149249] pinctrl core: initialized pinctrl subsystem [ 0.150113] [ 0.150399] ************************************************************* [ 0.152007] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.153006] ** ** [ 0.154006] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.156012] ** ** [ 0.159011] ** This means that this kernel is built to expose internal ** [ 0.160008] ** IOMMU data structures, which may compromise security on ** [ 0.162007] ** your system. ** [ 0.163006] ** ** [ 0.164008] ** If you see this message and you are not debugging the ** [ 0.166008] ** kernel, report this immediately to your vendor! ** [ 0.167005] ** ** [ 0.169009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.170006] ************************************************************* [ 0.171511] NET: Registered protocol family 16 [ 0.173441] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.175038] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.176033] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.179082] cpuidle: using governor menu [ 0.180499] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.181357] PCI: Using configuration type 1 for base access [ 0.183092] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.189037] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.190031] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.192100] cryptd: max_cpu_qlen set to 1000 [ 0.194190] ACPI: Added _OSI(Module Device) [ 0.195017] ACPI: Added _OSI(Processor Device) [ 0.196008] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.196900] ACPI: Added _OSI(Processor Aggregator Device) [ 0.200095] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.205297] ACPI: Interpreter enabled [ 0.206040] ACPI: PM: (supports S0 S3 S4 S5) [ 0.207009] ACPI: Using IOAPIC for interrupt routing [ 0.208099] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.211332] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.218053] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.219025] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.221011] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.224062] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.228371] acpiphp: Slot [2] registered [ 0.229103] acpiphp: Slot [5] registered [ 0.231101] acpiphp: Slot [6] registered [ 0.232120] acpiphp: Slot [7] registered [ 0.234113] acpiphp: Slot [8] registered [ 0.235099] acpiphp: Slot [9] registered [ 0.237109] acpiphp: Slot [10] registered [ 0.239089] acpiphp: Slot [3] registered [ 0.240078] acpiphp: Slot [4] registered [ 0.241087] acpiphp: Slot [11] registered [ 0.242099] acpiphp: Slot [12] registered [ 0.242942] acpiphp: Slot [13] registered [ 0.244073] acpiphp: Slot [14] registered [ 0.245090] acpiphp: Slot [15] registered [ 0.245972] acpiphp: Slot [16] registered [ 0.247052] acpiphp: Slot [17] registered [ 0.247974] acpiphp: Slot [18] registered [ 0.248070] acpiphp: Slot [19] registered [ 0.248925] acpiphp: Slot [20] registered [ 0.250057] acpiphp: Slot [21] registered [ 0.251017] acpiphp: Slot [22] registered [ 0.251910] acpiphp: Slot [23] registered [ 0.253091] acpiphp: Slot [24] registered [ 0.254081] acpiphp: Slot [25] registered [ 0.256093] acpiphp: Slot [26] registered [ 0.257177] acpiphp: Slot [27] registered [ 0.259102] acpiphp: Slot [28] registered [ 0.260138] acpiphp: Slot [29] registered [ 0.261102] acpiphp: Slot [30] registered [ 0.263095] acpiphp: Slot [31] registered [ 0.264094] PCI host bridge to bus 0000:00 [ 0.266022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.268028] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.271030] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.273019] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.275019] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.277018] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.279170] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.281812] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.284301] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.292680] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.300065] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.301011] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.303011] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.304009] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.305439] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.307708] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.310029] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.311662] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.315011] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.324880] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.328011] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.331879] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.343017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.349016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.376018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.385880] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.400016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.406016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.428130] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.442955] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.451015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.463021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.486018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.505255] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.511016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.521015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.534016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.544847] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.551014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.558014] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.573026] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.584942] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.591016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.597017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.610018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.620036] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.623422] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.627266] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.630282] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.632202] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.637146] iommu: Default domain type: Passthrough [ 0.639485] SCSI subsystem initialized [ 0.641140] ACPI: bus type USB registered [ 0.642091] usbcore: registered new interface driver usbfs [ 0.644108] usbcore: registered new interface driver hub [ 0.646077] usbcore: registered new device driver usb [ 0.648218] pps_core: LinuxPPS API ver. 1 registered [ 0.650024] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.652056] PTP clock support registered [ 0.654118] EDAC MC: Ver: 3.0.0 [ 0.656524] PCI: Using ACPI for IRQ routing [ 0.658767] NetLabel: Initializing [ 0.659009] NetLabel: domain hash size = 128 [ 0.661009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.663091] NetLabel: unlabeled traffic allowed by default [ 0.665096] vgaarb: loaded [ 0.667218] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.668013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.675257] clocksource: Switched to clocksource kvm-clock [ 0.779890] VFS: Disk quotas dquot_6.6.0 [ 0.781360] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.784122] *** VALIDATE ramfs *** [ 0.785537] *** VALIDATE hugetlbfs *** [ 0.787382] pnp: PnP ACPI init [ 0.789872] pnp: PnP ACPI: found 6 devices [ 0.829132] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.832460] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.835322] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.837851] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.841644] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.845254] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.849369] NET: Registered protocol family 2 [ 0.852645] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.859348] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.864407] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.870845] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.874168] TCP: Hash tables configured (established 65536 bind 65536) [ 0.877199] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.880384] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.883379] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.886237] NET: Registered protocol family 1 [ 0.889279] RPC: Registered named UNIX socket transport module. [ 0.891561] RPC: Registered udp transport module. [ 0.893636] RPC: Registered tcp transport module. [ 0.895570] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.897581] NET: Registered protocol family 44 [ 0.899225] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.901060] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.902659] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.905064] PCI: CLS 0 bytes, default 64 [ 0.906307] Unpacking initramfs... [ 2.270693] debug: unmapping init [mem 0xffffa1007cc54000-0xffffa1007ffbffff] [ 2.274193] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.276605] software IO TLB: mapped [mem 0x00000000a2600000-0x00000000a6600000] (64MB) [ 2.279802] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.742293] Initialise system trusted keyrings [ 2.743961] Key type blacklist registered [ 2.745526] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.755529] zbud: loaded [ 2.758948] *** VALIDATE nfs *** [ 2.759852] *** VALIDATE nfs4 *** [ 2.760920] pstore: using deflate compression [ 2.765072] Platform Keyring initialized [ 2.862279] NET: Registered protocol family 38 [ 2.864218] Key type asymmetric registered [ 2.866219] Asymmetric key parser 'x509' registered [ 2.867886] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.870134] io scheduler mq-deadline registered [ 2.871141] io scheduler kyber registered [ 2.872108] io scheduler bfq registered [ 2.873365] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.875487] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.877203] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.878816] ACPI: Power Button [PWRF] [ 2.883559] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.890372] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.902107] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.908450] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.927504] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.955134] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.982118] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.986912] Non-volatile memory driver v1.3 [ 2.988496] Linux agpgart interface v0.103 [ 3.021630] virtio_blk virtio1: [vda] 145896 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.024175] vda: detected capacity change from 0 to 74698752 [ 3.037226] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.040134] vdb: detected capacity change from 0 to 1073741824 [ 3.054920] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.059392] vdc: detected capacity change from 0 to 2621440000 [ 3.078407] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.081368] vdd: detected capacity change from 0 to 2621440000 [ 3.097971] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.101210] vde: detected capacity change from 0 to 4294967296 [ 3.123454] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.126265] vdf: detected capacity change from 0 to 4294967296 [ 3.134684] libphy: Fixed MDIO Bus: probed [ 3.141849] usbcore: registered new interface driver usbserial_generic [ 3.144119] usbserial: USB Serial support registered for generic [ 3.146374] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.150330] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.152298] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.154958] mousedev: PS/2 mouse device common for all mice [ 3.157877] rtc_cmos 00:05: RTC can wake from S4 [ 3.161362] rtc_cmos 00:05: registered as rtc0 [ 3.163416] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.167306] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.169934] intel_pstate: CPU model not supported [ 3.172894] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.178098] hid: raw HID events driver (C) Jiri Kosina [ 3.178128] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.179978] usbcore: registered new interface driver usbhid [ 3.184031] usbhid: USB HID core driver [ 3.185471] drop_monitor: Initializing network drop monitor service [ 3.187415] Initializing XFRM netlink socket [ 3.189310] NET: Registered protocol family 10 [ 3.191702] Segment Routing with IPv6 [ 3.192936] NET: Registered protocol family 17 [ 3.194716] mpls_gso: MPLS GSO support [ 3.199996] RAS: Correctable Errors collector initialized. [ 3.202319] AVX version of gcm_enc/dec engaged. [ 3.204627] AES CTR mode by8 optimization enabled [ 3.272431] sched_clock: Marking stable (3272411624, 0)->(4090534892, -818123268) [ 3.275696] registered taskstats version 1 [ 3.277573] Loading compiled-in X.509 certificates [ 3.279492] zswap: loaded using pool lzo/zbud [ 3.301109] Key type big_key registered [ 3.314223] Key type encrypted registered [ 3.315438] ima: No TPM chip found, activating TPM-bypass! [ 3.317227] ima: Allocated hash algorithm: sha1 [ 3.319291] ima: No architecture policies found [ 3.321091] evm: Initialising EVM extended attributes: [ 3.323130] evm: security.selinux [ 3.324388] evm: security.ima [ 3.325644] evm: security.capability [ 3.327156] evm: HMAC attrs: 0x1 [ 3.330050] rtc_cmos 00:05: setting system clock to 2026-09-03 13:28:52 UTC (1788442132) [ 3.336218] debug: unmapping init [mem 0xffffffff97803000-0xffffffff979fffff] [ 3.339737] debug: unmapping init [mem 0xffffffff96582000-0xffffffff96858fff] [ 3.349067] Write protecting the kernel read-only data: 28672k [ 3.352158] debug: unmapping init [mem 0xffffffff94c03000-0xffffffff94dfffff] [ 3.355068] debug: unmapping init [mem 0xffffffff95514000-0xffffffff955fffff] [ 3.389768] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.398150] systemd[1]: Detected virtualization kvm. [ 3.400913] systemd[1]: Detected architecture x86-64. [ 3.403385] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.428249] systemd[1]: No hostname configured. [ 3.430664] systemd[1]: Set hostname to . [ 3.433463] random: systemd: uninitialized urandom read (16 bytes read) [ 3.436933] systemd[1]: Initializing machine ID from random generator. [ 3.482059] random: ln: uninitialized urandom read (6 bytes read) [ 3.544972] random: systemd: uninitialized urandom read (16 bytes read) [ 3.547744] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.553062] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.557433] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Initrd Root Device. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.082669] device-mapper: uevent: version 1.0.3 [ 4.084836] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.771338] virtio_net virtio0 ens2: renamed from eth0 [ 4.776365] random: fast init done [ 4.869752] scsi host0: ata_piix [ 4.968474] scsi host1: ata_piix [ 4.982750] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.984925] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.282266] dracut-initqueue[582]: RTNETLINK answers: File exists [ 9.845902] random: crng init done [ 9.851528] random: 7 urandom warning(s) missed due to ratelimiting [ 11.422662] hrtimer: interrupt took 3652350 ns Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 12.078205] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 15.976060] printk: systemd: 26 output lines suppressed due to ratelimiting [ 17.208192] SELinux: Disabled at runtime. [ 17.333617] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 17.365222] systemd[1]: Detected virtualization kvm. [ 17.366486] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 19.099440] systemd[1]: initrd-switch-root.service: Succeeded. [ 19.107773] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 19.152591] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 19.166927] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 19.175816] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 19.207186] systemd[1]: Starting Journal Service... Starting Journal Service... [ 19.219689] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Mounting POSIX Message Queue File System... Mounting Huge Pages File System... [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Remount Root and Kernel File Systems... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Initrd File Systems. Starting Apply Kernel Variables... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-serial\x2dgetty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target RPC Port Mapper. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [[ 20.023920] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS  OK ] Reached target Paths. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 21.203742] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 22.158801] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 22.413014] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 22.850206] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 22.949704] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit)[ 27.605422] Key type dns_resolver registered [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit)[ 28.575271] NFS: Registering the id_resolver key type [ 28.577475] Key type id_resolver registered [ 28.579548] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Load/Save Random Seed. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. [ OK ] Reached target sshd-keygen.target. Starting Network Manager... Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg257-server login: [ 95.946127] libcfs: loading out-of-tree module taints kernel. [ 96.061752] Key type ._llcrypt registered [ 96.068845] Key type .llcrypt registered [ 96.195100] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_hostid [ 119.914771] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 121.863921] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 121.878139] alg: No test for adler32 (adler32-zlib) [ 123.465190] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 124.588954] LNet: Added LNI 192.168.202.157@tcp [8/256/0/180] [ 126.464196] Key type lgssc registered [ 128.699199] Lustre: Echo OBD driver; http://www.lustre.org/ [ 147.982356] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 194.236450] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 209.338911] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 209.388659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 210.818928] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 210.860803] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 210.926595] Lustre: lustre-MDT0000: new disk, initializing [ 211.066882] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 211.083744] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 216.042504] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 231.253230] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 231.424857] Lustre: 6510:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 231.492608] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 231.497620] Lustre: Skipped 1 previous similar message [ 231.595923] Lustre: lustre-MDT0001: new disk, initializing [ 231.692258] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 231.731647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 231.746260] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 237.037749] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 242.297196] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 254.096928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 254.639995] Lustre: lustre-OST0000: new disk, initializing [ 254.649240] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 254.667473] Lustre: 8448:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 254.788564] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 262.528768] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 264.258906] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 264.285412] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 264.369295] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 279.952610] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 280.134788] Lustre: lustre-OST0001: new disk, initializing [ 280.145668] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 280.165447] Lustre: 9521:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 280.322618] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 287.135970] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 289.305924] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 289.326775] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 289.482504] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 299.142900] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 306.790469] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 314.549993] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing check_logdir /tmp/testlogs/ [ 321.074577] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing yml_node [ 327.372628] Lustre: DEBUG MARKER: Client: 2.17.56.51 [ 330.638577] Lustre: DEBUG MARKER: MDS: 2.17.56.51 [ 333.888719] Lustre: DEBUG MARKER: OSS: 2.17.56.51 [ 335.773187] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 3 09:34:23 EDT 2026 [ 354.912332] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 357.135176] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 375.266241] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 375.279146] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 375.297828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 376.802171] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 376.805960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 376.816430] Lustre: Skipped 1 previous similar message [ 379.973230] Lustre: server umount lustre-MDT0000 complete [ 383.557712] LustreError: 6501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442512 with bad export cookie 8296156543154389178 [ 383.558647] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 383.565033] LustreError: 6501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 387.041061] LustreError: 6516:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 387.041363] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 387.057515] LustreError: 6516:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 387.057825] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 387.057830] Lustre: Skipped 1 previous similar message [ 387.108369] Lustre: Skipped 3 previous similar messages [ 388.078785] LustreError: 6517:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 388.092790] LustreError: 6517:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 390.103814] Lustre: server umount lustre-MDT0001 complete [ 394.144089] Lustre: server umount lustre-OST0000 complete [ 397.738620] Lustre: server umount lustre-OST0001 complete [ 404.943625] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_hostid [ 414.963666] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 452.285465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 452.516847] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 452.551772] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 452.639706] Lustre: lustre-MDT0000: new disk, initializing [ 452.731557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 452.767267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 458.266936] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 469.852078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 469.949637] Lustre: 15077:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 469.995979] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 469.999060] Lustre: Skipped 1 previous similar message [ 470.062503] Lustre: lustre-MDT0001: new disk, initializing [ 470.190697] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 470.247833] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 470.261436] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 475.461306] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 485.712204] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 491.506743] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 499.008477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 499.359153] Lustre: lustre-OST0000: new disk, initializing [ 499.362985] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 499.374281] Lustre: 17044:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 499.502256] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 500.873497] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 500.892524] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 501.050764] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 506.753473] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 517.939648] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 523.452244] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 523.741589] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 529.448231] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 529.690100] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 537.066698] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 537.068609] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 537.075840] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 537.075849] Lustre: Skipped 3 previous similar messages [ 537.081746] Lustre: Skipped 1 previous similar message [ 542.179332] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 542.201379] Lustre: Skipped 2 previous similar messages [ 546.272136] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 546.630126] Lustre: server umount lustre-OST0000 complete [ 552.420302] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 552.428408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 552.436811] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 552.441317] LustreError: Skipped 1 previous similar message [ 565.217691] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 565.866874] Lustre: server umount lustre-MDT0000 complete [ 567.780584] LustreError: 15088:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 567.809854] LustreError: 15088:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 570.992113] LustreError: 15069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442700 with bad export cookie 8296156543154390711 [ 570.993917] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 571.008170] LustreError: 15069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 571.509473] Lustre: server umount lustre-MDT0001 complete [ 576.738379] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 09:38:23 (1788442703) [ 585.317598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 585.746137] LustreError: 19315:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 585.829320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 590.823165] LustreError: 19316:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 590.969516] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 600.042447] LustreError: 19316:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 600.090677] LustreError: 19316:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 600.169521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 605.551829] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 614.417758] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 620.977268] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 629.514313] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 637.045617] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 645.253608] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 655.329267] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 655.340939] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 655.371895] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 655.373576] Lustre: Skipped 2 previous similar messages [ 655.406575] Lustre: Skipped 6 previous similar messages [ 658.805899] Lustre: server umount lustre-OST0000 complete [ 677.344861] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 677.588981] Lustre: server umount lustre-MDT0000 complete [ 680.929511] LustreError: 19315:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 680.945154] LustreError: 19315:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 681.776475] LustreError: 19296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442810 with bad export cookie 8296156543154391593 [ 681.782979] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 681.787162] LustreError: 19296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 682.303926] Lustre: server umount lustre-MDT0001 complete [ 695.098272] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [ 697.999618] Key type lgssc unregistered [ 698.247992] LNet: 22854:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 698.265633] LNetError: 22854:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 699.301929] LNet: Removed LNI 192.168.202.157@tcp [ 700.252416] Key type .llcrypt unregistered [ 700.254315] Key type ._llcrypt unregistered [ 727.141321] Key type ._llcrypt registered [ 727.144340] Key type .llcrypt registered [ 727.259464] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 728.483106] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 728.561604] alg: No test for adler32 (adler32-zlib) [ 729.717316] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 729.984666] LNet: Added LNI 192.168.202.157@tcp [8/256/0/180] [ 731.696257] Key type lgssc registered [ 733.115715] Lustre: Echo OBD driver; http://www.lustre.org/ [ 741.705779] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 741.738197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 743.454581] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 748.722796] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 753.635971] LustreError: 24273:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 753.649428] LustreError: 24273:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 757.977273] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 758.438510] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 762.673470] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 770.270901] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 775.942403] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 783.642720] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 784.008536] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 788.159962] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 791.299435] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 799.877588] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 819.172087] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 819.172176] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 819.188949] LustreError: Skipped 1 previous similar message [ 819.191701] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 819.214023] Lustre: Skipped 1 previous similar message [ 819.237831] Lustre: Skipped 1 previous similar message [ 823.523568] Lustre: server umount lustre-OST0000 complete [ 829.414640] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 829.444216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 829.451138] Lustre: Skipped 1 previous similar message [ 834.532947] LustreError: 24273:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 834.578397] Lustre: server umount lustre-MDT0000 complete [ 838.395029] LustreError: 24254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442967 with bad export cookie 10409573383383492897 [ 838.411315] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 838.672620] Lustre: server umount lustre-MDT0001 complete [ 851.469514] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [ 854.587419] Key type lgssc unregistered [ 854.933716] LNet: 28254:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 854.940989] LNetError: 28254:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 855.986749] LNet: Removed LNI 192.168.202.157@tcp [ 856.948507] Key type .llcrypt unregistered [ 856.951034] Key type ._llcrypt unregistered [ 881.361153] Key type ._llcrypt registered [ 881.365573] Key type .llcrypt registered [ 881.530205] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 882.615222] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 882.767089] alg: No test for adler32 (adler32-zlib) [ 883.743741] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 883.913782] LNet: Added LNI 192.168.202.157@tcp [8/256/0/180] [ 885.544168] Key type lgssc registered [ 886.815873] Lustre: Echo OBD driver; http://www.lustre.org/ [ 896.482856] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 896.503505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 898.125240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 904.222955] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 908.268913] LustreError: 29667:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 908.290948] LustreError: 29667:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 913.379990] LustreError: 29668:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 914.175949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 914.719410] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 919.879892] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 928.818565] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 935.619691] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 952.844176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 953.403308] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 962.545201] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 968.764611] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 971.219298] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 978.914354] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 978.927531] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 978.931399] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 978.950309] Lustre: Skipped 1 previous similar message [ 984.038729] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 984.048432] Lustre: Skipped 1 previous similar message [ 984.800627] Lustre: server umount lustre-OST0000 complete [ 989.154660] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 989.164951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 989.175923] Lustre: Skipped 1 previous similar message [ 989.199889] Lustre: Skipped 2 previous similar messages [ 994.274126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 994.278025] Lustre: Skipped 1 previous similar message [ 995.266616] Lustre: server umount lustre-MDT0000 complete [ 999.399507] LustreError: 29667:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 999.440456] LustreError: 29667:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1000.117851] LustreError: 29647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443129 with bad export cookie 3770703439034009039 [ 1000.127604] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1000.134629] LustreError: 29647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1000.490908] Lustre: server umount lustre-MDT0001 complete [ 1013.326838] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [ 1017.542571] Key type lgssc unregistered [ 1017.885610] LNet: 33551:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1017.898414] LNetError: 33551:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1017.928897] LNet: Removed LNI 192.168.202.157@tcp [ 1019.183639] Key type .llcrypt unregistered [ 1019.186725] Key type ._llcrypt unregistered [ 1045.426730] Key type ._llcrypt registered [ 1045.429044] Key type .llcrypt registered [ 1045.510763] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 1046.607619] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1046.650258] alg: No test for adler32 (adler32-zlib) [ 1047.714880] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1047.986244] LNet: Added LNI 192.168.202.157@tcp [8/256/0/180] [ 1049.712290] Key type lgssc registered [ 1050.686161] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1062.144075] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1062.163645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1063.553596] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1069.805517] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 1073.127266] LustreError: 34970:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1073.145047] LustreError: 34970:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1078.243612] LustreError: 34971:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1080.669780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1080.913035] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1086.471806] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 1094.099699] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1099.711172] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1108.123533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1108.608630] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1114.113365] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1115.704590] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 1125.626947] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1136.612346] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1136.618181] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1136.638710] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1138.518428] Lustre: server umount lustre-OST0000 complete [ 1146.856099] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1146.858892] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1146.865667] Lustre: Skipped 2 previous similar messages [ 1146.868651] Lustre: Skipped 1 previous similar message [ 1148.766765] Lustre: server umount lustre-MDT0000 complete [ 1151.969432] LustreError: 34971:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1152.008040] LustreError: 34971:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1152.460604] LustreError: 36815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443281 with bad export cookie 1457429286531231501 [ 1152.470697] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1152.475154] LustreError: 36815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1152.932451] Lustre: server umount lustre-MDT0001 complete [ 1165.516266] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [ 1168.659297] Key type lgssc unregistered [ 1168.930927] LNet: 38508:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1168.941555] LNetError: 38508:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1168.960724] LNet: Removed LNI 192.168.202.157@tcp [ 1169.744410] Key type .llcrypt unregistered [ 1169.748050] Key type ._llcrypt unregistered [ 1195.983410] Key type ._llcrypt registered [ 1195.989166] Key type .llcrypt registered [ 1196.173513] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 1197.190390] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1197.234118] alg: No test for adler32 (adler32-zlib) [ 1198.542486] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1198.804785] LNet: Added LNI 192.168.202.157@tcp [8/256/0/180] [ 1200.544228] Key type lgssc registered [ 1202.162182] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1209.748802] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1209.767456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1211.299491] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1216.774031] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 1221.605552] LustreError: 39921:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1221.621897] LustreError: 39921:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1226.089533] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1226.576686] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1231.438388] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 1240.567210] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1248.258957] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1256.666693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1256.912155] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1264.396864] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 1273.847612] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1281.600095] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1291.747784] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1291.751416] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1291.761424] LustreError: Skipped 1 previous similar message [ 1291.785305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1294.868356] Lustre: lustre-OST0000: Not available for connect from 192.168.202.57@tcp (stopping) [ 1294.880313] Lustre: Skipped 1 previous similar message [ 1296.444908] Lustre: server umount lustre-OST0000 complete [ 1318.883262] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1318.904373] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1318.924715] Lustre: Skipped 1 previous similar message [ 1318.935489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1322.467720] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1322.471669] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1324.321300] Lustre: server umount lustre-MDT0000 complete [ 1327.586746] LustreError: 40924:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1327.620367] LustreError: 40924:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1328.257444] LustreError: 39902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443457 with bad export cookie 4163247897411982187 [ 1328.266089] LustreError: 39902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1328.266663] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1334.709440] Lustre: server umount lustre-MDT0001 complete [ 1347.951485] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [ 1351.038463] Key type lgssc unregistered [ 1351.313836] LNet: 43656:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1351.329035] LNetError: 43656:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1351.345969] LNet: Removed LNI 192.168.202.157@tcp [ 1352.430917] Key type .llcrypt unregistered [ 1352.435955] Key type ._llcrypt unregistered [ 1378.380683] Key type ._llcrypt registered [ 1378.387608] Key type .llcrypt registered [ 1378.557492] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 1380.211900] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1380.309641] alg: No test for adler32 (adler32-zlib) [ 1381.401554] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1381.579322] LNet: Added LNI 192.168.202.157@tcp [8/256/0/180] [ 1383.257086] Key type lgssc registered [ 1384.805766] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1392.838690] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1392.855455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1394.235260] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1397.667511] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 1404.396427] LustreError: 45083:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1404.428957] LustreError: 45083:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1406.324143] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1406.766734] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1411.171365] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 1418.928622] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1424.783785] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1431.664220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1432.066573] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1439.233360] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1440.304242] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 1450.464432] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1458.677045] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.57@tcp (stopping) [ 1461.731469] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1461.735961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1461.757485] Lustre: Skipped 2 previous similar messages [ 1461.773495] Lustre: Skipped 2 previous similar messages [ 1463.777164] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.57@tcp (stopping) [ 1464.664826] Lustre: server umount lustre-MDT0000 complete [ 1466.860655] LustreError: 45080:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1466.882885] LustreError: 45080:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1468.773926] LustreError: 45064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443597 with bad export cookie 17266087614590540659 [ 1468.788853] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1468.895621] LustreError: 45078:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1468.926856] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.57@tcp (stopping) [ 1471.989849] LustreError: 46306:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1471.994514] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1474.020787] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1474.034568] Lustre: Skipped 2 previous similar messages [ 1478.123137] LustreError: 45080:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1478.143452] LustreError: 45080:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1483.232204] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1483.234581] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1483.253278] Lustre: Skipped 2 previous similar messages [ 1483.419188] Lustre: server umount lustre-MDT0001 complete [ 1498.608844] Lustre: server umount lustre-OST0000 complete [ 1517.754472] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [ 1520.077348] Key type lgssc unregistered [ 1520.453429] LNet: 48951:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1520.475682] LNetError: 48951:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1521.513391] LNet: Removed LNI 192.168.202.157@tcp [ 1522.382252] Key type .llcrypt unregistered [ 1522.385099] Key type ._llcrypt unregistered [ 1549.773396] Key type ._llcrypt registered [ 1549.775124] Key type .llcrypt registered [ 1549.903533] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 1551.304538] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1551.370270] alg: No test for adler32 (adler32-zlib) [ 1552.499188] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1552.746514] LNet: Added LNI 192.168.202.157@tcp [8/256/0/180] [ 1554.408210] Key type lgssc registered [ 1555.557867] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1563.870681] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1563.886847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1579.488647] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1584.609249] LustreError: 50352:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.157@tcp: failed processing log, type 4: rc = -110 [ 1611.424282] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1618.375637] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 1631.377276] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1798.013819] Lustre: server umount lustre-OST0000 complete [ 1818.935067] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [ 1822.044303] Key type lgssc unregistered [ 1822.381184] LNet: 52032:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1822.405651] LNetError: 52032:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1823.463739] LNet: Removed LNI 192.168.202.157@tcp [ 1824.506448] Key type .llcrypt unregistered [ 1824.513320] Key type ._llcrypt unregistered [ 1854.090274] Key type ._llcrypt registered [ 1854.092100] Key type .llcrypt registered [ 1854.215639] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 1855.330053] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1855.372137] alg: No test for adler32 (adler32-zlib) [ 1856.720116] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1857.149770] LNet: Added LNI 192.168.202.157@tcp [8/256/0/180] [ 1858.896462] Key type lgssc registered [ 1860.499596] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1868.986621] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1869.024702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1870.590984] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1875.955528] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 1880.034970] LustreError: 53447:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1880.054923] LustreError: 53447:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1884.741319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1885.154642] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1890.082117] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 1898.794538] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1905.128222] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1913.092310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1913.416778] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1915.507995] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1919.938638] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 1929.562260] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1936.366676] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1936.368049] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1936.387282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1936.401251] Lustre: Skipped 1 previous similar message [ 1936.434121] Lustre: Skipped 1 previous similar message [ 1939.602670] Lustre: server umount lustre-OST0000 complete [ 1946.595737] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1946.595777] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1946.610329] LustreError: Skipped 1 previous similar message [ 1946.652966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1951.713261] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1951.725961] Lustre: Skipped 2 previous similar messages [ 1956.837419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1956.846678] Lustre: Skipped 1 previous similar message [ 1958.368175] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1958.634597] Lustre: server umount lustre-MDT0000 complete [ 1961.956408] LustreError: 53443:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1961.990513] LustreError: 53443:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1963.844214] LustreError: 53427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444093 with bad export cookie 2151930183579023406 [ 1963.849831] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1963.865132] LustreError: 53427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1964.509702] Lustre: server umount lustre-MDT0001 complete [ 1977.273856] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [ 1980.444599] Key type lgssc unregistered [ 1980.795891] LNet: 56997:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1980.807918] LNetError: 56997:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1980.834198] LNet: Removed LNI 192.168.202.157@tcp [ 1981.733662] Key type .llcrypt unregistered [ 1981.741817] Key type ._llcrypt unregistered [ 2009.606795] Key type ._llcrypt registered [ 2009.609985] Key type .llcrypt registered [ 2009.722903] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 2011.127313] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2011.195873] alg: No test for adler32 (adler32-zlib) [ 2012.356154] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2012.634352] LNet: Added LNI 192.168.202.157@tcp [8/256/0/180] [ 2014.400837] Key type lgssc registered [ 2015.488980] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2023.154919] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2023.166795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2038.688569] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2043.808814] LustreError: 58383:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.157@tcp: failed processing log, type 4: rc = -110 [ 2070.560190] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2078.665744] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 2087.310444] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2095.606447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2096.190190] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2096.270709] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2101.219532] LustreError: 58408:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2101.241060] LustreError: 58408:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2102.454734] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 2105.322331] LustreError: 59420:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2105.372372] LustreError: 59420:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2110.436851] LustreError: 59801:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2110.473221] LustreError: 59801:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2114.494452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2114.911490] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2120.259101] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 2127.703083] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2134.298341] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2137.059579] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2137.064641] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2137.072876] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2140.645559] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2140.647449] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2140.648085] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2142.747339] Lustre: server umount lustre-OST0000 complete [ 2170.339994] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2170.343995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2170.353107] Lustre: Skipped 1 previous similar message [ 2170.372919] Lustre: Skipped 2 previous similar messages [ 2175.458841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2180.577487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2180.581717] Lustre: Skipped 1 previous similar message [ 2182.113043] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2182.475983] Lustre: server umount lustre-MDT0000 complete [ 2185.700450] LustreError: 59424:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2185.730486] LustreError: 59424:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2186.094206] LustreError: 58390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444315 with bad export cookie 487323278440239866 [ 2186.106904] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2192.569705] Lustre: server umount lustre-MDT0001 complete [ 2205.226411] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [ 2209.333292] Key type lgssc unregistered [ 2209.646127] LNet: 62133:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2209.655428] LNetError: 62133:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2210.729495] LNet: Removed LNI 192.168.202.157@tcp [ 2211.536490] Key type .llcrypt unregistered [ 2211.539794] Key type ._llcrypt unregistered [ 2238.556691] Key type ._llcrypt registered [ 2238.558561] Key type .llcrypt registered [ 2238.846321] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 2240.136489] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2240.262672] alg: No test for adler32 (adler32-zlib) [ 2241.853989] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2242.323436] LNet: Added LNI 192.168.202.157@tcp [8/256/0/180] [ 2244.072199] Key type lgssc registered [ 2245.677779] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2255.013354] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2255.076137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2256.721479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2262.672298] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 2266.106032] LustreError: 63548:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2266.119772] LustreError: 63548:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2271.209397] LustreError: 63549:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2273.264381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2273.715254] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2278.248428] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 2286.864814] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2294.104466] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2300.160913] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2300.504592] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2301.572056] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2306.549549] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 2314.947094] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2333.665665] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2333.672371] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2333.689151] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2333.714071] Lustre: Skipped 1 previous similar message [ 2333.735321] Lustre: Skipped 1 previous similar message [ 2338.791574] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2338.802394] Lustre: Skipped 1 previous similar message [ 2339.823296] Lustre: server umount lustre-OST0000 complete [ 2345.444094] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2345.454710] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2345.479263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2349.027135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2349.029685] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2354.159862] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2354.164816] Lustre: Skipped 2 previous similar messages [ 2359.271059] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2359.539563] Lustre: server umount lustre-MDT0000 complete [ 2363.832487] LustreError: 63530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444492 with bad export cookie 12569744502333872149 [ 2363.842538] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2363.843577] LustreError: 63530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2364.106128] Lustre: server umount lustre-MDT0001 complete [ 2375.721202] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [ 2378.511001] Key type lgssc unregistered [ 2378.941048] LNet: 67109:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2378.966247] LNetError: 67109:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2378.996743] LNet: Removed LNI 192.168.202.157@tcp [ 2379.844751] Key type .llcrypt unregistered [ 2379.847464] Key type ._llcrypt unregistered [ 2407.395335] Key type ._llcrypt registered [ 2407.398079] Key type .llcrypt registered [ 2407.560926] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 2408.712716] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2408.828930] alg: No test for adler32 (adler32-zlib) [ 2409.873306] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2410.035373] LNet: Added LNI 192.168.202.157@tcp [8/256/0/180] [ 2411.664993] Key type lgssc registered [ 2412.823481] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2422.353662] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 10:09:10 (1788444550) [ 2429.370988] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2429.385175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2430.751544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2436.250600] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 2440.163807] LustreError: 68815:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2440.175550] LustreError: 68815:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2445.287104] LustreError: 68814:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2446.452920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2446.803504] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2452.344804] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 2461.323600] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2467.800324] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2474.691574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2475.170658] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2476.218742] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2482.362961] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 2491.139273] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2497.663667] Lustre: Failing over lustre-MDT0000 [ 2497.993494] Lustre: server umount lustre-MDT0000 complete [ 2498.022136] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2498.028808] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2498.038754] LustreError: 68815:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2502.115027] LustreError: 68814:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2502.125411] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2502.153239] LustreError: 68814:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2507.235633] LustreError: 68811:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2507.260894] LustreError: 68811:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 2508.421920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2508.602765] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2508.748075] Lustre: *** cfs_fail_loc=135, val=0*** [ 2508.838897] LustreError: 71903:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2508.846420] LustreError: 71903:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.157@tcp: cfg command failed: rc = -2 [ 2508.859792] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2508.859792] [ 2508.875076] LustreError: MGC192.168.202.157@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2508.907361] LustreError: 71891:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2508.917662] LustreError: 71891:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2508.924867] LustreError: 71891:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2508.931781] LustreError: 71891:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2508.949527] Lustre: server umount lustre-MDT0000 complete [ 2508.953506] LustreError: 71891:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2515.464741] LustreError: 68809:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2515.489037] LustreError: 68809:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 2516.827477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2517.037721] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2517.052483] LustreError: 68806:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2517.386619] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2517.445108] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2520.548201] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2522.612974] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2522.654909] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 2522.687314] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 2522.728456] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2532.848448] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2532.859793] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2532.865851] Lustre: Skipped 1 previous similar message [ 2532.876104] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2536.097387] Lustre: server umount lustre-OST0000 complete [ 2543.078779] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2543.079138] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2543.080654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2543.080661] Lustre: Skipped 1 previous similar message [ 2543.116769] Lustre: Skipped 2 previous similar messages [ 2546.510176] Lustre: server umount lustre-MDT0000 complete [ 2548.201963] LustreError: 70075:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2548.222705] LustreError: 70075:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2550.584686] LustreError: 68795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444679 with bad export cookie 7600658329753296803 [ 2550.595309] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2550.598889] LustreError: 68795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2550.860658] Lustre: server umount lustre-MDT0001 complete [ 2564.235416] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [ 2568.205033] Key type lgssc unregistered [ 2568.548658] LNet: 73685:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2568.558986] LNetError: 73685:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2568.578938] LNet: Removed LNI 192.168.202.157@tcp [ 2569.922565] Key type .llcrypt unregistered [ 2569.927911] Key type ._llcrypt unregistered [ 2594.804325] Key type ._llcrypt registered [ 2594.809324] Key type .llcrypt registered [ 2594.886562] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 2595.922688] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2596.164682] alg: No test for adler32 (adler32-zlib) [ 2597.275963] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2597.480575] LNet: Added LNI 192.168.202.157@tcp [8/256/0/180] [ 2599.257703] Key type lgssc registered [ 2600.356992] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2607.553456] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2607.571614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2608.855728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2612.885678] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 2619.366458] LustreError: 75099:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2619.393742] LustreError: 75099:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2622.850190] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2623.260779] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2627.504595] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 2637.124714] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2644.178087] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2652.051793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2652.416234] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2654.521983] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2659.629803] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 2668.797318] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2676.029117] Lustre: Failing over lustre-MDT0000 [ 2676.316594] Lustre: server umount lustre-MDT0000 complete [ 2676.748644] LustreError: 76118:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2679.278660] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2679.295281] Lustre: Skipped 1 previous similar message [ 2679.305882] LustreError: 75095:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2679.338568] LustreError: 75095:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2681.819958] LustreError: 76118:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2686.951289] LustreError: 75095:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2686.975916] LustreError: 75095:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2687.179576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2687.363355] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2687.641341] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2687.840740] LustreError: 78184:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2687.856707] LustreError: 78184:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.157@tcp: cfg command failed: rc = -9 [ 2687.875348] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2687.875348] [ 2687.886623] LustreError: MGC192.168.202.157@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2687.900249] LustreError: 78173:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2687.907141] LustreError: 78173:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 2687.923426] LustreError: 78173:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2687.933595] LustreError: 78173:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2687.965771] Lustre: server umount lustre-MDT0000 complete [ 2687.971136] LustreError: 78173:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2695.146976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2695.264873] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2695.274931] LustreError: 78531:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGC192.168.202.157@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (00000000f73ed393) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 2695.299895] LustreError: 75091:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2695.627183] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2695.687308] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2698.224201] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2700.812318] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2700.845909] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 2700.917129] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2701.032306] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 2711.016966] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2711.036673] Lustre: Skipped 2 previous similar messages [ 2711.037895] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2711.051581] Lustre: Skipped 1 previous similar message [ 2716.143386] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2716.162844] Lustre: Skipped 1 previous similar message [ 2716.985540] Lustre: server umount lustre-OST0000 complete [ 2726.370814] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2726.374266] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2726.376332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2726.382963] Lustre: Skipped 1 previous similar message [ 2726.399793] Lustre: Skipped 1 previous similar message [ 2727.976609] Lustre: server umount lustre-MDT0000 complete [ 2731.498811] LustreError: 75099:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2731.532555] LustreError: 75099:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 2732.504247] LustreError: 75833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444861 with bad export cookie 10232580004355729407 [ 2732.514633] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2732.517566] LustreError: 75833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2732.794564] Lustre: server umount lustre-MDT0001 complete [ 2745.560596] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [ 2748.250851] Key type lgssc unregistered [ 2748.516378] LNet: 79970:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2748.530605] LNetError: 79970:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2748.554943] LNet: Removed LNI 192.168.202.157@tcp [ 2749.418426] Key type .llcrypt unregistered [ 2749.420162] Key type ._llcrypt unregistered [ 2764.727707] Key type ._llcrypt registered [ 2764.730244] Key type .llcrypt registered [ 2764.848709] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [ 2765.203467] Key type .llcrypt unregistered [ 2765.205542] Key type ._llcrypt unregistered [ 2792.255925] Key type ._llcrypt registered [ 2792.257782] Key type .llcrypt registered [ 2792.466475] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 2793.742533] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2794.057427] alg: No test for adler32 (adler32-zlib) [ 2795.555808] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2795.912078] LNet: Added LNI 192.168.202.157@tcp [8/256/0/180] [ 2797.576321] Key type lgssc registered [ 2799.112216] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2810.096932] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2810.178536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2812.084501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2818.628953] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 2822.628105] LustreError: 81880:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2822.640510] LustreError: 81880:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2827.774979] LustreError: 81881:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2831.406454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2831.743839] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2836.629299] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 2844.762917] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2851.702764] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2859.063737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2859.594131] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2868.746909] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 2878.266744] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2882.131318] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2892.268776] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2892.289878] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2892.302433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2892.303656] Lustre: Skipped 1 previous similar message [ 2892.314568] Lustre: Skipped 1 previous similar message [ 2897.390151] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2897.405651] Lustre: Skipped 1 previous similar message [ 2898.378412] Lustre: server umount lustre-OST0000 complete [ 2903.521428] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2903.527866] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2903.545293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2907.619976] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2907.620373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2907.670333] Lustre: Skipped 1 previous similar message [ 2912.741360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2912.755056] Lustre: Skipped 1 previous similar message [ 2917.346165] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2917.642866] Lustre: server umount lustre-MDT0000 complete [ 2917.862768] LustreError: 82609:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2917.883779] LustreError: 82609:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2921.961908] LustreError: 81876:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2921.987813] LustreError: 81876:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2922.331553] LustreError: 81860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445051 with bad export cookie 34964618809190110 [ 2922.337968] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2922.346986] LustreError: 81860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2922.945107] Lustre: server umount lustre-MDT0001 complete [ 2934.686488] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [ 2937.612694] Key type lgssc unregistered [ 2937.928962] LNet: 85443:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2937.935774] LNetError: 85443:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2937.951421] LNet: Removed LNI 192.168.202.157@tcp [ 2938.740611] Key type .llcrypt unregistered [ 2938.743937] Key type ._llcrypt unregistered [ 2962.793673] Key type ._llcrypt registered [ 2962.796794] Key type .llcrypt registered [ 2962.904515] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 2963.789060] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2963.837395] alg: No test for adler32 (adler32-zlib) [ 2964.847724] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2965.045888] LNet: Added LNI 192.168.202.157@tcp [8/256/0/180] [ 2966.689871] Key type lgssc registered [ 2968.200486] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2976.793190] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2976.844203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2978.464642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2983.475717] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 2988.003473] LustreError: 86855:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2988.042641] LustreError: 86855:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2992.378402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2992.716213] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2997.813134] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 3008.252965] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3017.092760] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3024.541562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3024.759795] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3031.666578] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 3040.792302] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3048.329803] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 3061.294909] Lustre: server umount lustre-OST0000 complete [ 3068.903508] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3068.914522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3068.924672] Lustre: Skipped 1 previous similar message [ 3068.946554] Lustre: Skipped 1 previous similar message [ 3074.017715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3074.028795] Lustre: Skipped 1 previous similar message [ 3075.150809] Lustre: server umount lustre-MDT0000 complete [ 3078.112761] Lustre: 86169:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788445191/real 1788445191] req@00000000a717a59f x1875320602918144/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788445207 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3078.149865] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3078.171444] LustreError: 88698:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3078.204585] LustreError: 88698:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3079.137664] Lustre: 86166:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788445192/real 1788445192] req@00000000bd9d825a x1875320602918912/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788445208 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3079.881636] LustreError: 86836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445209 with bad export cookie 11873121229404460749 [ 3079.886492] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3079.899753] LustreError: 86836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3080.373389] Lustre: server umount lustre-MDT0001 complete [ 3091.664316] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [ 3094.704562] Key type lgssc unregistered [ 3095.071042] LNet: 90416:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3095.089504] LNetError: 90416:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3095.109904] LNet: Removed LNI 192.168.202.157@tcp [ 3096.050779] Key type .llcrypt unregistered [ 3096.053345] Key type ._llcrypt unregistered [ 3120.606386] Key type ._llcrypt registered [ 3120.610972] Key type .llcrypt registered [ 3120.844974] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 3122.286303] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3122.437959] alg: No test for adler32 (adler32-zlib) [ 3124.031423] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3124.255637] LNet: Added LNI 192.168.202.157@tcp [8/256/0/180] [ 3125.928302] Key type lgssc registered [ 3127.704619] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3138.183804] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3138.275758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3140.003381] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3145.952865] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 3150.322422] LustreError: 91832:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3150.332875] LustreError: 91832:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3155.427478] LustreError: 91831:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3156.837625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3157.517094] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3163.721322] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 3173.325279] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3180.413088] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3188.370084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3188.947171] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3195.357569] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 3203.096223] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3210.486607] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 3228.143381] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3228.150567] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3228.166077] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3231.203973] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3231.223308] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3231.225032] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3231.250496] Lustre: Skipped 1 previous similar message [ 3232.447673] Lustre: server umount lustre-OST0000 complete [ 3239.392954] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3239.399062] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3239.407338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3241.442907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3241.446246] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3243.958449] Lustre: server umount lustre-MDT0000 complete [ 3246.565386] LustreError: 91827:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3246.598931] LustreError: 91827:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3248.739623] LustreError: 91811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445377 with bad export cookie 6907804346333400749 [ 3248.756287] LustreError: 91811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3248.758545] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3249.154368] Lustre: server umount lustre-MDT0001 complete [ 3261.374519] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [ 3264.341317] Key type lgssc unregistered [ 3264.677303] LNet: 95392:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3264.695992] LNetError: 95392:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3265.767529] LNet: Removed LNI 192.168.202.157@tcp [ 3266.816665] Key type .llcrypt unregistered [ 3266.820666] Key type ._llcrypt unregistered [ 3291.999481] Key type ._llcrypt registered [ 3292.003911] Key type .llcrypt registered [ 3292.102637] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 3293.121907] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3293.247744] alg: No test for adler32 (adler32-zlib) [ 3294.468082] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3294.662359] LNet: Added LNI 192.168.202.157@tcp [8/256/0/180] [ 3296.368169] Key type lgssc registered [ 3297.243843] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3306.315466] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3306.336437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3321.889329] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3327.076393] LustreError: 96779:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.157@tcp: failed processing log, type 4: rc = -110 [ 3353.760249] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3362.001676] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 3370.990933] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3377.732922] Lustre: server umount lustre-OST0000 complete [ 3390.123907] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 10:25:17 (1788445517) [ 3398.732226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3399.299961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3404.205027] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 3404.776847] LustreError: 98419:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3404.789089] LustreError: 98419:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3408.873883] LustreError: 98418:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3413.844387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3414.000663] LustreError: 98419:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3414.314205] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3419.596910] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 3428.272828] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3435.378365] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3443.725891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3444.141641] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3445.189031] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3452.381829] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 3460.908987] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3475.950191] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3475.954133] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3475.981645] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3480.036667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3480.041525] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3480.054604] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3481.815225] Lustre: server umount lustre-OST0000 complete [ 3490.281902] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3490.318366] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3490.330226] Lustre: Skipped 2 previous similar messages [ 3492.915316] Lustre: server umount lustre-MDT0000 complete [ 3495.404083] LustreError: 98414:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3495.415858] LustreError: 98414:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3497.397500] LustreError: 98401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445626 with bad export cookie 2219225040506314068 [ 3497.418931] LustreError: 98401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3497.421822] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3498.022902] Lustre: server umount lustre-MDT0001 complete [ 3510.898629] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [ 3514.193686] Key type lgssc unregistered [ 3514.500544] LNet: 102118:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3514.509672] LNetError: 102118:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3514.520225] LNet: Removed LNI 192.168.202.157@tcp [ 3515.456520] Key type .llcrypt unregistered [ 3515.458161] Key type ._llcrypt unregistered [ 3564.798855] Key type ._llcrypt registered [ 3564.801841] Key type .llcrypt registered [ 3564.977178] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 3566.858654] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3566.948731] alg: No test for adler32 (adler32-zlib) [ 3568.131842] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3568.385316] LNet: Added LNI 192.168.202.157@tcp [8/256/0/180] [ 3570.096462] Key type lgssc registered [ 3571.368637] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3581.878495] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3581.890738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3583.365491] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3589.515907] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 3593.698029] LustreError: 104306:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3593.719818] LustreError: 104306:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3598.818325] LustreError: 104305:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3599.470089] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3599.964080] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3605.360713] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 3614.334833] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3620.858410] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3627.935964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3628.335977] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3629.405690] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3634.568529] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 3641.656775] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3656.174723] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3656.179996] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3656.189289] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3656.219245] Lustre: Skipped 1 previous similar message [ 3661.290359] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3661.298504] Lustre: Skipped 1 previous similar message [ 3662.152840] Lustre: server umount lustre-OST0000 complete [ 3666.406439] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3666.411312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3666.416684] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3666.420686] Lustre: Skipped 1 previous similar message [ 3666.449207] Lustre: Skipped 2 previous similar messages [ 3671.526456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3671.548046] Lustre: Skipped 1 previous similar message [ 3672.868388] Lustre: server umount lustre-MDT0000 complete [ 3676.648293] LustreError: 104306:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3676.689872] LustreError: 104306:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3678.459696] LustreError: 104287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445807 with bad export cookie 3170221897426557049 [ 3678.476175] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3678.478160] LustreError: 104287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3679.231107] Lustre: server umount lustre-MDT0001 complete [ 3692.169490] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [ 3695.552309] Key type lgssc unregistered [ 3695.909235] LNet: 107863:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3695.924374] LNetError: 107863:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3695.942384] LNet: Removed LNI 192.168.202.157@tcp [ 3696.761956] Key type .llcrypt unregistered [ 3696.766346] Key type ._llcrypt unregistered [ 3716.253876] Key type ._llcrypt registered [ 3716.255777] Key type .llcrypt registered [ 3716.395937] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 3717.555290] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3717.576540] alg: No test for adler32 (adler32-zlib) [ 3718.631734] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3718.884114] LNet: Added LNI 192.168.202.157@tcp [8/256/0/180] [ 3720.681291] Key type lgssc registered [ 3722.201770] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3730.117670] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3730.146583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3745.696572] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3750.816721] LustreError: 109105:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.157@tcp: failed processing log, type 4: rc = -110 [ 3777.440286] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3782.759611] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 3792.711736] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3801.632852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3802.190119] LustreError: MGC192.168.202.157@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3802.206862] LustreError: 110130:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3802.219472] LustreError: 110130:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3802.235988] LustreError: 110130:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3802.360161] Lustre: server umount lustre-MDT0000 complete [ 3802.362310] LustreError: 110130:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3821.539695] Lustre: 108443:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788445934/real 1788445934] req@0000000042b1d4d4 x1875321393515904/t0(0) o400->MGC192.168.202.157@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788445950 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3821.586516] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3821.601755] LustreError: 109117:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3842.098484] Lustre: server umount lustre-OST0000 complete [ 3854.898981] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_hostid [ 3869.285779] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 3907.419758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3907.717049] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3907.770480] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3907.887505] Lustre: lustre-MDT0000: new disk, initializing [ 3907.986711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3908.023137] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3913.366875] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 3925.866099] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3926.064641] Lustre: 112927:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3926.147423] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3926.151945] Lustre: Skipped 1 previous similar message [ 3926.330817] Lustre: lustre-MDT0001: new disk, initializing [ 3926.516836] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3926.578692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3926.600321] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3932.121220] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 3943.180655] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3948.737861] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3956.179430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3956.596460] Lustre: lustre-OST0000: new disk, initializing [ 3956.601452] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3956.608883] Lustre: 114894:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3956.698717] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3962.052282] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3962.078566] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3962.293190] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3964.537825] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 3976.764752] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3983.827922] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3984.185437] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3991.392534] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3991.540725] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3998.177279] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3998.182578] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3998.193380] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3998.228157] Lustre: Skipped 1 previous similar message [ 4000.324981] Lustre: server umount lustre-OST0000 complete [ 4008.428578] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4008.435496] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4008.447508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4008.447515] Lustre: Skipped 1 previous similar message [ 4008.467137] LustreError: Skipped 1 previous similar message [ 4010.997897] Lustre: server umount lustre-MDT0000 complete [ 4015.194715] LustreError: 112918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446144 with bad export cookie 5030227453359461579 [ 4015.203074] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4015.208263] LustreError: 112918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4015.554995] Lustre: server umount lustre-MDT0001 complete [ 4025.559680] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 10:35:53 (1788446153) [ 4029.458979] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 4047.643636] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_hostid [ 4058.390860] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 4097.965818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4098.242524] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4098.286676] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4098.393267] Lustre: lustre-MDT0000: new disk, initializing [ 4098.502801] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4098.533504] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4103.102146] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 4114.325299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4114.487701] Lustre: 119800:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4114.548758] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4114.552888] Lustre: Skipped 1 previous similar message [ 4114.653712] Lustre: lustre-MDT0001: new disk, initializing [ 4114.751703] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4114.783730] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4114.803634] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4119.916815] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 4129.531887] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4136.105472] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4143.501403] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4143.779418] Lustre: lustre-OST0000: new disk, initializing [ 4143.783080] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4143.794666] Lustre: 121765:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4143.891932] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4150.372852] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 4153.932032] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4153.947907] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4154.112183] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4161.361759] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4167.296698] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4167.617996] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4172.966592] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4173.220967] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4179.938446] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4179.942488] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4179.951908] Lustre: Skipped 1 previous similar message [ 4179.958558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4179.978087] Lustre: Skipped 2 previous similar messages [ 4181.909749] Lustre: server umount lustre-OST0000 complete [ 4186.593486] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4186.597888] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4186.608736] Lustre: Skipped 1 previous similar message [ 4186.622559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4192.281340] Lustre: server umount lustre-MDT0000 complete [ 4195.307459] LustreError: 119810:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4195.330289] LustreError: 119810:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 4196.610753] LustreError: 119790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446325 with bad export cookie 5030227453359462454 [ 4196.613207] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4196.624446] LustreError: 119790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4196.945836] Lustre: server umount lustre-MDT0001 complete [ 4205.536829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4206.117729] LustreError: 123984:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4206.252996] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4211.149552] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 4211.690274] LustreError: 123985:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4216.803099] LustreError: 123984:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4221.655185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4221.924947] LustreError: 123985:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4228.005884] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 4237.155919] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4244.228714] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4252.541285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4260.959697] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 4270.017908] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4288.372927] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 4297.191649] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4297.195941] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4297.224046] Lustre: Skipped 1 previous similar message [ 4297.238997] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4297.247309] Lustre: Skipped 3 previous similar messages [ 4299.401758] Lustre: server umount lustre-OST0000 complete [ 4307.427916] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4307.430101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4307.448468] Lustre: Skipped 2 previous similar messages [ 4307.460773] Lustre: Skipped 2 previous similar messages [ 4310.291761] Lustre: server umount lustre-MDT0000 complete [ 4312.555474] LustreError: 123981:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4312.582138] LustreError: 123981:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4314.183109] LustreError: 126885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446443 with bad export cookie 5030227453359463343 [ 4314.192903] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4314.198070] LustreError: 126885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4314.482473] Lustre: server umount lustre-MDT0001 complete [ 4327.352944] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [ 4331.848945] Key type lgssc unregistered [ 4332.377659] LNet: 127641:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4332.385190] LNetError: 127641:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4332.406652] LNet: Removed LNI 192.168.202.157@tcp [ 4333.660306] Key type .llcrypt unregistered [ 4333.662738] Key type ._llcrypt unregistered [ 4355.975629] Key type ._llcrypt registered [ 4355.977246] Key type .llcrypt registered [ 4356.057246] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_hostid [ 4370.762935] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 4372.454755] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4372.542787] alg: No test for adler32 (adler32-zlib) [ 4373.660638] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 4373.936853] LNet: Added LNI 192.168.202.157@tcp [8/256/0/180] [ 4375.640236] Key type lgssc registered [ 4376.840303] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4415.468560] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4415.505406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4416.769335] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4416.811879] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4416.986992] Lustre: lustre-MDT0000: new disk, initializing [ 4417.104956] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4417.156653] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4422.496568] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 4434.311720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4434.459653] Lustre: 130798:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4434.543718] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4434.549966] Lustre: Skipped 1 previous similar message [ 4434.744130] Lustre: lustre-MDT0001: new disk, initializing [ 4434.920816] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4434.973760] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4434.994011] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4440.066605] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 4450.226744] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4456.860235] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4465.167613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4465.546100] Lustre: lustre-OST0000: new disk, initializing [ 4465.557182] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4465.569246] Lustre: 132766:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4465.865734] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4471.894678] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4471.914479] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4472.120738] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4473.021141] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 4484.040395] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4489.090349] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4489.379624] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4495.189290] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4495.513927] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4503.010395] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4503.024951] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4503.034324] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4503.041088] Lustre: Skipped 1 previous similar message [ 4503.051607] Lustre: Skipped 1 previous similar message [ 4508.138931] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4508.146592] Lustre: Skipped 1 previous similar message [ 4512.224553] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4512.442538] Lustre: server umount lustre-OST0000 complete [ 4516.834361] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4516.837778] LustreError: Skipped 1 previous similar message [ 4516.843350] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4516.859964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4518.379813] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4522.349375] Lustre: server umount lustre-MDT0000 complete [ 4526.378242] LustreError: 130791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446655 with bad export cookie 4250248701499968847 [ 4526.389912] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4526.391051] LustreError: 130791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4526.812354] Lustre: server umount lustre-MDT0001 complete [ 4537.345526] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 10:44:25 (1788446665) [ 4545.868496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4546.425834] LustreError: 135182:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4546.452987] LustreError: 135182:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4546.561994] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4551.649643] LustreError: 135183:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4552.377062] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 4556.783063] LustreError: 135183:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4561.911230] LustreError: 135182:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4562.203840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4562.704943] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4567.919978] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 4576.452784] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4583.263400] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4587.490904] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4587.492024] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4587.522119] Lustre: Skipped 1 previous similar message [ 4587.541380] Lustre: Skipped 3 previous similar messages [ 4592.212914] Lustre: server umount lustre-MDT0000 complete [ 4593.635260] LustreError: 135178:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4593.683033] LustreError: 135178:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4596.942134] LustreError: 135164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446726 with bad export cookie 4250248701499969729 [ 4596.955583] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4596.956225] LustreError: 135164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4597.394764] Lustre: server umount lustre-MDT0001 complete [ 4606.464098] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 10:45:34 (1788446734) [ 4613.489372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4629.153723] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4634.272428] LustreError: 137598:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.157@tcp: failed processing log, type 4: rc = -110 [ 4660.000310] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4667.981512] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 4675.730913] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4678.095632] Lustre: server umount lustre-OST0000 complete [ 4686.877692] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 10:46:54 (1788446814) [ 4693.954853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4694.451027] LustreError: 139045:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4694.533283] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4699.470128] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 4709.767845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4716.340903] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 4724.599734] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4731.708792] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4739.593859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4740.000476] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4740.011584] Lustre: Skipped 1 previous similar message [ 4746.033703] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 4753.659183] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4774.886733] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4774.891170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4774.900431] Lustre: Skipped 2 previous similar messages [ 4774.923731] Lustre: Skipped 4 previous similar messages [ 4778.797744] Lustre: server umount lustre-MDT0000 complete [ 4780.011423] LustreError: 139046:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4780.031182] LustreError: 139046:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 4783.413283] LustreError: 141649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446912 with bad export cookie 4250248701499970394 [ 4783.417400] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4783.426509] LustreError: 141649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4783.858065] Lustre: server umount lustre-MDT0001 complete [ 4798.442691] Lustre: server umount lustre-OST0000 complete [ 4808.034904] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 10:48:55 (1788446935) [ 4815.137853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4815.480794] LustreError: 142724:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4815.544769] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4820.060742] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 4828.903360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4833.357349] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 4842.253451] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4848.171555] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4855.505137] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4859.896364] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4862.491158] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 4869.717545] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4874.470344] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4874.705190] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4879.357840] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4879.612782] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4884.455280] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4884.462835] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4884.490523] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4884.493241] Lustre: Skipped 1 previous similar message [ 4887.983081] Lustre: server umount lustre-OST0000 complete [ 4898.252302] Lustre: server umount lustre-MDT0000 complete [ 4899.811536] LustreError: 142725:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4899.824305] LustreError: 142725:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4902.433441] LustreError: 142706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447031 with bad export cookie 4250248701499971465 [ 4902.440129] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4902.449099] LustreError: 142706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4902.820284] Lustre: server umount lustre-MDT0001 complete [ 4912.972526] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 10:50:40 (1788447040) [ 4920.192114] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4935.904604] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4941.024552] LustreError: 146891:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.157@tcp: failed processing log, type 4: rc = -110 [ 4966.752388] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4966.761468] Lustre: Skipped 2 previous similar messages [ 4973.982358] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 4983.110863] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4990.967032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4991.582311] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4996.450446] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 5005.763769] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5011.265644] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 5018.851762] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5026.358092] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5031.177326] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5031.364878] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5035.407196] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5035.669156] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5042.160198] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5042.165048] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5042.172648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5042.180652] Lustre: Skipped 4 previous similar messages [ 5042.188931] Lustre: Skipped 4 previous similar messages [ 5043.858901] Lustre: server umount lustre-MDT0000 complete [ 5047.282321] LustreError: 147929:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5047.308894] LustreError: 147929:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 5048.099655] LustreError: 146897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447177 with bad export cookie 4250248701499972487 [ 5048.113216] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5048.120819] LustreError: 146897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5048.413405] Lustre: server umount lustre-MDT0001 complete [ 5063.096238] Lustre: server umount lustre-OST0000 complete [ 5072.852609] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 10:53:20 (1788447200) [ 5081.061475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5096.674530] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5101.792598] LustreError: 151090:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.157@tcp: failed processing log, type 4: rc = -110 [ 5134.764389] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 5142.211801] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5148.712543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5149.291591] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 5153.759599] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 5162.361057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5167.070081] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 5174.537556] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5180.949956] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5187.693625] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5187.992272] Lustre: lustre-OST0001: new disk, initializing [ 5187.995900] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5188.006350] Lustre: 153962:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5194.296707] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5194.306580] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5194.428915] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5194.508478] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 5203.574073] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5208.051740] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5208.252604] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5213.726382] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5214.054521] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5219.298932] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5219.317753] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5219.347752] Lustre: Skipped 1 previous similar message [ 5219.363340] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5222.858982] Lustre: server umount lustre-OST0000 complete [ 5230.563311] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5230.587348] LustreError: Skipped 1 previous similar message [ 5233.916652] Lustre: server umount lustre-OST0001 complete [ 5244.157223] Lustre: server umount lustre-MDT0000 complete [ 5248.635569] LustreError: 151096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447377 with bad export cookie 4250248701499973439 [ 5248.646926] LustreError: 151096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5248.655258] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5249.053329] Lustre: server umount lustre-MDT0001 complete [ 5278.168987] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 10:56:45 (1788447405) [ 5280.025961] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 5281.613097] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 5284.170402] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 10:56:51 (1788447411) [ 5292.060945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5292.443857] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5292.461816] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5292.667532] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5292.670757] Lustre: Skipped 6 previous similar messages [ 5298.703290] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 5308.457768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5308.525327] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5308.544507] Lustre: 157524:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5312.701494] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 5321.311271] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5327.504673] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5334.113399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5334.348144] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5341.929693] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 5342.108587] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 5350.864507] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5356.071097] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5356.295983] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5361.616733] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5361.940815] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5367.778211] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5367.779360] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5367.788948] LustreError: Skipped 1 previous similar message [ 5367.790979] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5367.790987] Lustre: Skipped 7 previous similar messages [ 5367.826218] Lustre: Skipped 6 previous similar messages [ 5370.675744] Lustre: server umount lustre-OST0000 complete [ 5393.785176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5401.915806] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 5408.692708] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5408.718371] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5408.720423] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5408.741067] Lustre: Skipped 1 previous similar message [ 5410.453680] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5418.496382] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5419.152968] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5427.084118] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5427.434318] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5435.467914] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8e69c4cf2000.ost_server_uuid 50 [ 5437.278907] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8e69c4cf2000.ost_server_uuid in IDLE state after 0 sec [ 5437.444991] LustreError: 157531:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.202.57@tcp) returned error from blocking AST (req@00000000f04d9012 x1875322080600064 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000b30c9502/0x3afbeaacdbf8a899 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.57@tcp remote: 0x10b60a9f559daebb expref: 5 pid: 157530 timeout: 5537 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5437.462086] LustreError: lustre-MDT0000: A client on nid 192.168.202.57@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5437.467125] LustreError: 157519:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.202.57@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000b30c9502/0x3afbeaacdbf8a899 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.57@tcp remote: 0x10b60a9f559daebb expref: 6 pid: 157530 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5443.050532] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5447.320375] Lustre: server umount lustre-OST0000 complete [ 5457.450451] Lustre: server umount lustre-MDT0000 complete [ 5459.938690] LustreError: 157534:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5459.961233] LustreError: 157534:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 5461.665453] LustreError: 157516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447590 with bad export cookie 4250248701499974622 [ 5461.672282] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5461.676170] LustreError: 157516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5468.337867] Lustre: server umount lustre-MDT0001 complete [ 5482.064705] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [ 5485.519120] Key type lgssc unregistered [ 5485.884217] LNet: 163644:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5485.891435] LNetError: 163644:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5485.934668] LNet: Removed LNI 192.168.202.157@tcp [ 5487.614432] Key type .llcrypt unregistered [ 5487.616641] Key type ._llcrypt unregistered [ 5518.445993] Key type ._llcrypt registered [ 5518.454216] Key type .llcrypt registered [ 5518.605523] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 5520.144984] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5520.333632] alg: No test for adler32 (adler32-zlib) [ 5521.370905] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 5521.648804] LNet: Added LNI 192.168.202.157@tcp [8/256/0/180] [ 5523.440218] Key type lgssc registered [ 5524.517123] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5534.283580] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5534.306982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5536.310034] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5543.441336] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 5546.473501] LustreError: 165071:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5546.495613] LustreError: 165071:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5551.597546] LustreError: 165072:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5553.386392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5553.957258] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5557.386916] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 5567.666939] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5574.971756] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5583.485272] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5583.837437] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5591.350598] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 5599.576419] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5606.385344] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5612.360448] Lustre: Failing over lustre-MDT0000 [ 5612.701388] Lustre: server umount lustre-MDT0000 complete [ 5615.587020] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5615.592344] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5615.609497] LustreError: 165071:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5616.616443] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5616.644148] Lustre: Skipped 1 previous similar message [ 5621.735852] LustreError: 167368:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5621.768619] LustreError: 167368:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 5625.016916] LustreError: 165051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.57@tcp arrived at 1788447754 with bad export cookie 7478825954625356846 [ 5625.030374] LustreError: 165051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5626.864867] LustreError: 165802:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5626.878540] LustreError: 165802:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5631.968371] Lustre: 164370:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788447745/real 1788447745] req@0000000041c309c5 x1875323283076864/t0(0) o400->MGC192.168.202.157@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788447761 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5631.992500] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5633.677404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5637.098597] LustreError: 165072:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5637.125212] LustreError: 165072:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 5642.557096] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5642.652970] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5647.606560] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 5647.845930] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5647.856665] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5650.345601] Lustre: lustre-MDT0000: Denying connection for new client 5292ff36-3ab0-4628-962d-165b9cf9485b (at 192.168.202.57@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:07 [ 5659.627360] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5659.636644] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5659.651886] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5663.201845] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5663.205937] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5664.937244] Lustre: server umount lustre-OST0000 complete [ 5669.340061] LustreError: 168818:0:(ldlm_lib.c:3004:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5669.349884] Lustre: 168166:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5669.354197] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5669.368355] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5669.378447] LustreError: 168166:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000004b3bfbf7 x1875323283097600/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5669.387584] LustreError: 168166:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5669.393555] LustreError: 168166:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5669.400411] Lustre: lustre-MDT0000: Recovery over after 0:22, of 2 clients 0 recovered and 2 were evicted. [ 5669.685742] Lustre: server umount lustre-MDT0000 complete [ 5673.441366] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5673.463708] LustreError: 167368:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5673.477415] LustreError: 167368:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5673.491087] LustreError: 165053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447802 with bad export cookie 7478825954625357175 [ 5673.506116] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5673.511373] LustreError: 165053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5675.860662] Lustre: server umount lustre-MDT0001 complete [ 5689.741036] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [ 5693.338665] Key type lgssc unregistered [ 5693.582617] LNet: 169573:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5693.588356] LNetError: 169573:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5694.632808] LNet: Removed LNI 192.168.202.157@tcp [ 5695.592374] Key type .llcrypt unregistered [ 5695.594992] Key type ._llcrypt unregistered [ 5729.046601] Key type ._llcrypt registered [ 5729.049236] Key type .llcrypt registered [ 5729.157229] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 5730.530888] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5730.573069] alg: No test for adler32 (adler32-zlib) [ 5732.349474] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 5732.803381] LNet: Added LNI 192.168.202.157@tcp [8/256/0/180] [ 5734.633449] Key type lgssc registered [ 5736.579906] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5744.057909] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5744.078981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5745.813440] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5750.876334] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 5755.374962] LustreError: 170989:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5755.394377] LustreError: 170989:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5760.484131] LustreError: 170990:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5761.500965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5761.894446] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5769.353069] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 5779.503841] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5787.254064] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5796.744713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5797.205835] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5805.302867] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 5815.262389] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5815.720893] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5826.036992] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5826.038516] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5826.040899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5826.046452] Lustre: Skipped 1 previous similar message [ 5831.137836] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5831.143683] Lustre: Skipped 1 previous similar message [ 5836.263193] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5836.283292] Lustre: Skipped 2 previous similar messages [ 5838.304446] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5838.630522] Lustre: server umount lustre-OST0000 complete [ 5843.938024] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5843.954969] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5843.976060] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5846.500940] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5851.624239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5851.636229] Lustre: Skipped 3 previous similar messages [ 5858.273684] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5858.657488] Lustre: server umount lustre-MDT0000 complete [ 5861.864464] LustreError: 170984:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5861.875117] LustreError: 170984:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5862.523686] LustreError: 170970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447991 with bad export cookie 2421398888258959487 [ 5862.534387] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5862.544290] LustreError: 170970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5862.833916] Lustre: server umount lustre-MDT0001 complete [ 5876.635175] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [ 5880.315674] Key type lgssc unregistered [ 5880.569146] LNet: 174532:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5880.575711] LNetError: 174532:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5880.613993] LNet: Removed LNI 192.168.202.157@tcp [ 5881.736850] Key type .llcrypt unregistered [ 5881.741472] Key type ._llcrypt unregistered [ 5922.300243] Key type ._llcrypt registered [ 5922.302232] Key type .llcrypt registered [ 5922.497563] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 5923.255116] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5923.277598] alg: No test for adler32 (adler32-zlib) [ 5924.364494] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 5924.594377] LNet: Added LNI 192.168.202.157@tcp [8/256/0/180] [ 5926.369107] Key type lgssc registered [ 5927.618123] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5935.451566] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5935.530341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5937.259596] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5943.073666] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 5946.861716] LustreError: 176235:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5946.893123] LustreError: 176235:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5951.972337] LustreError: 176236:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5952.751866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5953.343667] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5958.897382] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 5966.400460] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5973.512553] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5982.030718] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5982.126089] LustreError: 178058:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5982.409733] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5987.842163] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5990.134339] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 6000.288836] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6008.291061] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6008.298256] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6008.305330] LustreError: Skipped 1 previous similar message [ 6008.305929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6008.320585] Lustre: Skipped 1 previous similar message [ 6008.339440] Lustre: Skipped 1 previous similar message [ 6013.247528] Lustre: server umount lustre-OST0000 complete [ 6018.533704] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6018.538911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6018.543061] Lustre: Skipped 1 previous similar message [ 6023.664578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6028.773054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6028.777340] Lustre: Skipped 2 previous similar messages [ 6032.352885] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6032.910468] Lustre: server umount lustre-MDT0000 complete [ 6033.900717] LustreError: 176230:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6033.931746] LustreError: 176230:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6039.012862] LustreError: 177188:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6039.049021] LustreError: 177188:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6039.451981] LustreError: 176216:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448168 with bad export cookie 222916536812252066 [ 6039.458366] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6039.494066] LustreError: 176216:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6040.142418] Lustre: server umount lustre-MDT0001 complete [ 6056.290232] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [ 6059.756511] Key type lgssc unregistered [ 6060.122301] LNet: 179777:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6060.134846] LNetError: 179777:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6060.152726] LNet: Removed LNI 192.168.202.157@tcp [ 6061.508996] Key type .llcrypt unregistered [ 6061.512311] Key type ._llcrypt unregistered [ 6086.464349] Key type ._llcrypt registered [ 6086.466396] Key type .llcrypt registered [ 6086.576523] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 6088.123259] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6088.299416] alg: No test for adler32 (adler32-zlib) [ 6089.477350] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 6089.792660] LNet: Added LNI 192.168.202.157@tcp [8/256/0/180] [ 6091.529078] Key type lgssc registered [ 6092.836754] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6104.475938] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6104.489761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6105.892592] Lustre: *** cfs_fail_loc=135, val=0*** [ 6105.979386] LustreError: 181305:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 6105.983442] LustreError: 181305:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.157@tcp: cfg command failed: rc = -2 [ 6105.991235] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 6105.991235] [ 6106.004291] LustreError: MGC192.168.202.157@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 6106.012204] LustreError: 181267:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 6106.024464] LustreError: 181267:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 6106.030350] LustreError: 181267:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 6106.037075] LustreError: 181267:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 6106.117350] Lustre: server umount lustre-MDT0000 complete [ 6106.121336] LustreError: 181267:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 6113.990851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6114.406376] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6117.699893] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 6124.513495] LustreError: 181676:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6124.543435] LustreError: 181676:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6126.225664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6126.653782] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6131.963611] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 6140.507185] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6146.491340] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6149.099169] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6149.119946] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6149.121643] Lustre: Skipped 1 previous similar message [ 6149.142386] Lustre: Skipped 1 previous similar message [ 6151.201704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6151.219221] Lustre: Skipped 1 previous similar message [ 6155.279884] Lustre: server umount lustre-MDT0000 complete [ 6156.258046] LustreError: 181676:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6156.291615] LustreError: 181676:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6159.551457] LustreError: 181657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448288 with bad export cookie 11047329305366017363 [ 6159.558102] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6159.562354] LustreError: 181657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6159.937793] Lustre: server umount lustre-MDT0001 complete [ 6172.969310] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [ 6176.598901] Key type lgssc unregistered [ 6176.983942] LNet: 184001:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6176.996761] LNetError: 184001:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6177.022230] LNet: Removed LNI 192.168.202.157@tcp [ 6178.094453] Key type .llcrypt unregistered [ 6178.096672] Key type ._llcrypt unregistered [ 6206.561875] Key type ._llcrypt registered [ 6206.568795] Key type .llcrypt registered [ 6206.696638] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [ 6207.152342] Key type .llcrypt unregistered [ 6207.154848] Key type ._llcrypt unregistered [ 6227.301707] Key type ._llcrypt registered [ 6227.313370] Key type .llcrypt registered [ 6227.499452] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 6229.499455] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6229.596216] alg: No test for adler32 (adler32-zlib) [ 6230.931489] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 6231.177121] LNet: Added LNI 192.168.202.157@tcp [8/256/0/180] [ 6232.856213] Key type lgssc registered [ 6234.449435] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6243.887164] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6243.918117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6259.552671] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6264.673657] LustreError: 186139:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.157@tcp: failed processing log, type 4: rc = -110 [ 6291.361400] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6299.126711] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 6308.196570] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6317.582724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6317.927349] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6318.063568] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 6323.169482] LustreError: 186165:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6323.211521] LustreError: 186165:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6325.694833] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 6326.264985] LustreError: 187177:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6326.295203] LustreError: 187177:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6331.377606] LustreError: 187177:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6331.412584] LustreError: 187177:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6336.482278] LustreError: 186165:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6336.513328] LustreError: 186165:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6338.186757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6338.772757] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6345.749588] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 6357.387898] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6363.985250] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6367.055162] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6379.489828] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6379.495842] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6379.499687] LustreError: Skipped 1 previous similar message [ 6379.500792] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6379.549163] Lustre: Skipped 1 previous similar message [ 6380.003517] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6380.012510] Lustre: Skipped 2 previous similar messages [ 6384.638765] Lustre: server umount lustre-OST0000 complete [ 6390.245505] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6390.251703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6390.261780] Lustre: Skipped 1 previous similar message [ 6390.286138] Lustre: Skipped 1 previous similar message [ 6394.837590] Lustre: server umount lustre-MDT0000 complete [ 6395.362207] LustreError: 188281:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6395.378549] LustreError: 188281:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6398.896046] LustreError: 186146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448528 with bad export cookie 17582495874485712879 [ 6398.913021] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6398.922104] LustreError: 186146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6399.341670] Lustre: server umount lustre-MDT0001 complete [ 6411.601634] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [ 6414.189447] Key type lgssc unregistered [ 6414.430732] LNet: 190112:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6414.441413] LNetError: 190112:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6414.460628] LNet: Removed LNI 192.168.202.157@tcp [ 6415.435092] Key type .llcrypt unregistered [ 6415.440574] Key type ._llcrypt unregistered [ 6442.275160] Key type ._llcrypt registered [ 6442.277830] Key type .llcrypt registered [ 6442.469125] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 6443.657232] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6443.668226] alg: No test for adler32 (adler32-zlib) [ 6444.869434] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 6445.147148] LNet: Added LNI 192.168.202.157@tcp [8/256/0/180] [ 6446.840358] Key type lgssc registered [ 6448.418650] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6458.896854] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6458.929797] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6460.438897] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6466.118491] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 6470.628325] LustreError: 191526:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6470.653599] LustreError: 191526:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6475.753083] LustreError: 191527:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6475.838195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6476.318411] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6481.044849] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 6490.456516] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6496.510230] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6503.808413] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6504.238453] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6505.294358] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 6512.274691] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 6522.484218] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6528.692525] Lustre: Failing over lustre-MDT0000 [ 6528.910501] Lustre: server umount lustre-MDT0000 complete [ 6529.549760] LustreError: 192532:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6532.087463] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6532.096853] LustreError: 192532:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6532.106317] Lustre: Skipped 1 previous similar message [ 6532.143845] LustreError: 192532:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6537.185405] LustreError: 191526:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6537.207145] LustreError: 191526:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6547.425801] LustreError: 191527:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6547.436906] LustreError: 191527:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 6548.448378] Lustre: 190835:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788448661/real 1788448661] req@0000000021234c9a x1875324251959808/t0(0) o400->MGC192.168.202.157@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788448677 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6548.485861] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6549.118955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6558.691450] LustreError: 190834:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000a4089a20 x1875324251965696/t0(0) o250->MGC192.168.202.157@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6559.083296] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6559.130885] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6560.224841] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6564.108165] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 6564.340684] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6564.363177] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 6564.432405] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 6569.245604] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6577.493779] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6589.939851] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6589.954721] Lustre: Skipped 1 previous similar message [ 6589.962882] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6589.972994] Lustre: Skipped 1 previous similar message [ 6595.053609] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6595.061819] Lustre: Skipped 1 previous similar message [ 6600.175922] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6600.192950] Lustre: Skipped 1 previous similar message [ 6603.232310] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6603.608372] Lustre: server umount lustre-OST0000 complete [ 6610.408661] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6610.416672] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6610.440652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6610.455678] Lustre: Skipped 2 previous similar messages [ 6610.471814] Lustre: Skipped 1 previous similar message [ 6614.241629] Lustre: server umount lustre-MDT0000 complete [ 6615.522450] LustreError: 191527:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6615.536145] LustreError: 191527:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 6618.191798] LustreError: 191506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448747 with bad export cookie 12009755611207281492 [ 6618.192696] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6618.202638] LustreError: 191506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6618.527739] Lustre: server umount lustre-MDT0001 complete [ 6630.801350] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [ 6633.685405] Key type lgssc unregistered [ 6633.997993] LNet: 196421:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6634.014310] LNetError: 196421:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6634.037488] LNet: Removed LNI 192.168.202.157@tcp [ 6635.064167] Key type .llcrypt unregistered [ 6635.069211] Key type ._llcrypt unregistered [ 6660.325362] Key type ._llcrypt registered [ 6660.327036] Key type .llcrypt registered [ 6660.415985] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 6661.339848] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6661.354496] alg: No test for adler32 (adler32-zlib) [ 6662.407629] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 6662.698951] LNet: Added LNI 192.168.202.157@tcp [8/256/0/180] [ 6664.464291] Key type lgssc registered [ 6665.480870] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6674.342042] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6674.354216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6675.984787] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6681.351453] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 6686.185403] LustreError: 197836:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6686.202716] LustreError: 197836:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6690.255658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6690.561924] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6695.766830] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 6702.610654] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6708.702694] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6715.977893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6716.678967] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6721.792958] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6723.675436] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 6731.761197] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6736.071759] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6740.609516] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6754.339731] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6762.982612] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6762.993100] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6763.016288] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6766.051030] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6766.053551] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6766.055093] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6766.063661] Lustre: Skipped 1 previous similar message [ 6768.093473] Lustre: server umount lustre-OST0000 complete [ 6772.707516] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6772.727911] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6772.740408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6776.290429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6776.295645] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6777.879789] Lustre: server umount lustre-MDT0000 complete [ 6781.409582] LustreError: 197833:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6781.440256] LustreError: 197833:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6781.619972] LustreError: 197817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448910 with bad export cookie 1500224650794478519 [ 6781.638434] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6781.641821] LustreError: 197817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6782.166334] Lustre: server umount lustre-MDT0001 complete [ 6794.422991] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [ 6797.627356] Key type lgssc unregistered [ 6798.079512] LNet: 201686:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6798.094441] LNetError: 201686:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6799.141776] LNet: Removed LNI 192.168.202.157@tcp [ 6800.454543] Key type .llcrypt unregistered [ 6800.456965] Key type ._llcrypt unregistered [ 6819.974355] Key type ._llcrypt registered [ 6819.976047] Key type .llcrypt registered [ 6820.070134] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 6821.772172] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6821.907268] alg: No test for adler32 (adler32-zlib) [ 6822.978702] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 6823.207074] LNet: Added LNI 192.168.202.157@tcp [8/256/0/180] [ 6824.928237] Key type lgssc registered [ 6826.310755] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6834.336993] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6834.354139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6835.664582] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6840.562221] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 6844.901661] LustreError: 202953:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6844.928888] LustreError: 202953:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6848.079027] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6848.516191] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6852.286185] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 6858.960439] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6864.477649] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6870.493199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6870.846808] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6877.000331] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 6880.045378] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6884.966678] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6890.570250] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6901.767642] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6910.713183] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6920.295586] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6929.378425] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6929.389429] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6929.395497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6931.449648] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6931.460761] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6931.463437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6931.479181] Lustre: Skipped 1 previous similar message [ 6934.483103] Lustre: server umount lustre-OST0000 complete [ 6940.643099] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6940.651172] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6940.676395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6944.828791] Lustre: server umount lustre-MDT0000 complete [ 6946.793314] LustreError: 203680:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6946.818509] LustreError: 203680:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6948.498455] LustreError: 202934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449077 with bad export cookie 9811176141597433298 [ 6948.500938] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6948.507949] LustreError: 202934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6948.807947] Lustre: server umount lustre-MDT0001 complete [ 6960.674268] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [ 6964.050268] Key type lgssc unregistered [ 6964.351906] LNet: 207877:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6964.365559] LNetError: 207877:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6964.388325] LNet: Removed LNI 192.168.202.157@tcp [ 6965.443481] Key type .llcrypt unregistered [ 6965.445226] Key type ._llcrypt unregistered [ 6994.128661] Key type ._llcrypt registered [ 6994.130182] Key type .llcrypt registered [ 6994.292663] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 6996.476706] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6996.585291] alg: No test for adler32 (adler32-zlib) [ 6997.756623] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 6998.004994] LNet: Added LNI 192.168.202.157@tcp [8/256/0/180] [ 6999.650365] Key type lgssc registered [ 7001.298329] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7010.589714] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7010.626205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7012.431603] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7017.332715] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 7022.563387] LustreError: 209290:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7022.596069] LustreError: 209290:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7027.690499] LustreError: 209291:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7028.759317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7029.397955] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7035.011394] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 7042.780544] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7049.513635] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7056.625768] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7057.033699] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7058.116452] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 7063.919038] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 7072.535308] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7089.124979] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7089.140172] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7089.159171] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7090.148515] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7090.156799] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7090.172845] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7093.650378] Lustre: server umount lustre-OST0000 complete [ 7100.394077] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7100.397165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7100.407558] Lustre: Skipped 1 previous similar message [ 7100.412628] Lustre: Skipped 1 previous similar message [ 7104.555989] Lustre: server umount lustre-MDT0000 complete [ 7105.506308] LustreError: 209287:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7105.520476] LustreError: 209287:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7109.657982] LustreError: 209272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449238 with bad export cookie 13867600154824475746 [ 7109.684251] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7109.688751] LustreError: 209272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7110.353274] Lustre: server umount lustre-MDT0001 complete [ 7124.001616] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [ 7126.203097] Key type lgssc unregistered [ 7126.428323] LNet: 212980:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7126.436799] LNetError: 212980:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7126.457303] LNet: Removed LNI 192.168.202.157@tcp [ 7127.102953] Key type .llcrypt unregistered [ 7127.104303] Key type ._llcrypt unregistered [ 7145.675331] Key type ._llcrypt registered [ 7145.678924] Key type .llcrypt registered [ 7145.769836] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 7146.704619] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7146.763656] alg: No test for adler32 (adler32-zlib) [ 7147.733761] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 7148.034523] LNet: Added LNI 192.168.202.157@tcp [8/256/0/180] [ 7149.736215] Key type lgssc registered [ 7150.757842] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7158.223684] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7158.279911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7160.010286] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7165.527971] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 7170.529804] LustreError: 214244:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7170.540873] LustreError: 214244:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7175.179331] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7175.780647] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7181.373869] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 7190.567475] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7198.099592] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7205.985167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7206.583315] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7213.433625] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 7222.867737] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7230.138364] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 7245.811552] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7245.813114] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7245.833026] Lustre: Skipped 1 previous similar message [ 7245.843651] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7250.918974] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7250.930601] Lustre: Skipped 2 previous similar messages [ 7251.224870] Lustre: server umount lustre-OST0000 complete [ 7257.569310] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7257.575861] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7257.587713] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7261.155360] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7261.164485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7261.193691] Lustre: Skipped 1 previous similar message [ 7262.522820] Lustre: server umount lustre-MDT0000 complete [ 7266.169212] LustreError: 214226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449395 with bad export cookie 10088611319190222561 [ 7266.172816] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7266.173645] LustreError: 214226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7266.387377] Lustre: server umount lustre-MDT0001 complete [ 7279.003770] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [ 7283.359812] Key type lgssc unregistered [ 7283.793922] LNet: 217954:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7283.817765] LNetError: 217954:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7283.860630] LNet: Removed LNI 192.168.202.157@tcp [ 7284.993864] Key type .llcrypt unregistered [ 7284.999038] Key type ._llcrypt unregistered [ 7307.961964] Key type ._llcrypt registered [ 7307.964562] Key type .llcrypt registered [ 7308.072593] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 7309.239094] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7309.272985] alg: No test for adler32 (adler32-zlib) [ 7310.433587] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 7310.858886] LNet: Added LNI 192.168.202.157@tcp [8/256/0/180] [ 7312.616246] Key type lgssc registered [ 7314.038348] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7323.401191] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7323.430268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7324.909103] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7330.738666] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 7335.400685] LustreError: 219221:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7335.425585] LustreError: 219221:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7340.521637] LustreError: 219222:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7341.103939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7341.646727] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7347.323874] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 7356.564325] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7363.774303] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7371.694832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7372.243946] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7379.221283] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 7389.245846] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7395.826726] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 7407.759098] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7407.857242] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7408.144806] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7417.002358] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 7428.900753] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7430.749788] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7442.955167] Lustre: Permanently deactivating lustre-OST0001 [ 7442.983538] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7450.201077] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 7458.915471] Lustre: Permanently reactivating lustre-OST0001 [ 7458.932269] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7458.940563] Lustre: Skipped 2 previous similar messages [ 7460.284458] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7460.303958] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7460.315068] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7460.341830] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7460.411900] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7471.076335] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7471.089305] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7471.100398] Lustre: Skipped 1 previous similar message [ 7471.110772] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7472.615320] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7472.617921] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7477.740711] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7477.743248] Lustre: Skipped 1 previous similar message [ 7482.859602] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7482.866795] Lustre: Skipped 1 previous similar message [ 7485.408170] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7485.934035] Lustre: server umount lustre-OST0001 complete [ 7487.986584] LustreError: 223139:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7488.002345] LustreError: 223139:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7493.098863] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7493.100706] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7493.103457] LustreError: 223139:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7493.108108] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7493.158473] Lustre: Skipped 1 previous similar message [ 7498.221375] LustreError: 221043:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7498.257979] LustreError: 221043:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7503.330686] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7503.341090] Lustre: Skipped 4 previous similar messages [ 7504.355594] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7504.536674] Lustre: server umount lustre-OST0000 complete [ 7510.497146] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7510.510574] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7523.296487] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7523.650489] Lustre: server umount lustre-MDT0000 complete [ 7523.818957] LustreError: 219216:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7523.854231] LustreError: 219216:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7527.814392] LustreError: 222082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449656 with bad export cookie 16247142427868394236 [ 7527.839184] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7527.852828] LustreError: 222082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7528.585824] Lustre: server umount lustre-MDT0001 complete [ 7542.031155] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [ 7544.838879] Key type lgssc unregistered [ 7545.174821] LNet: 224299:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7545.183085] LNetError: 224299:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7545.211594] LNet: Removed LNI 192.168.202.157@tcp [ 7546.194698] Key type .llcrypt unregistered [ 7546.199212] Key type ._llcrypt unregistered [ 7591.926284] Key type ._llcrypt registered [ 7591.929895] Key type .llcrypt registered [ 7592.076980] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 7593.629292] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7593.852164] alg: No test for adler32 (adler32-zlib) [ 7594.925684] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 7595.199249] LNet: Added LNI 192.168.202.157@tcp [8/256/0/180] [ 7596.896179] Key type lgssc registered [ 7598.178382] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7607.802252] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7607.817942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7609.126354] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7609.178623] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7609.374290] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7614.686197] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 7624.005516] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7624.131894] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7624.162532] Lustre: 226510:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7624.467024] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7628.868461] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 7637.706085] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7644.301852] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7651.953522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7652.213226] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7652.471538] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7659.839686] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 7661.286057] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 7669.439766] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7673.534814] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7679.092541] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7685.770940] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7692.381414] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7700.497800] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7725.548449] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7725.555443] Lustre: Skipped 2 previous similar messages [ 7763.582855] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7763.595581] Lustre: Skipped 4 previous similar messages [ 7827.979436] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7827.991295] Lustre: Skipped 7 previous similar messages [ 7850.981102] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7850.983055] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7850.988469] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7851.002031] Lustre: Skipped 1 previous similar message [ 7854.977568] Lustre: server umount lustre-OST0000 complete [ 7860.197616] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7860.204514] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7860.228680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7860.235524] Lustre: Skipped 1 previous similar message [ 7861.225070] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7861.233390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7861.238612] Lustre: Skipped 1 previous similar message [ 7865.920822] Lustre: server umount lustre-MDT0000 complete [ 7870.141669] LustreError: 226502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449999 with bad export cookie 5567416831412835190 [ 7870.145957] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7870.151285] LustreError: 226502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7870.548920] Lustre: server umount lustre-MDT0001 complete [ 7884.610561] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [ 7887.856803] Key type lgssc unregistered [ 7888.239851] LNet: 231333:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7888.250618] LNetError: 231333:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7888.274938] LNet: Removed LNI 192.168.202.157@tcp [ 7889.118379] Key type .llcrypt unregistered [ 7889.120295] Key type ._llcrypt unregistered [ 7913.171351] Key type ._llcrypt registered [ 7913.173255] Key type .llcrypt registered [ 7913.258465] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 7914.515445] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7914.535459] alg: No test for adler32 (adler32-zlib) [ 7915.584258] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 7915.821118] LNet: Added LNI 192.168.202.157@tcp [8/256/0/180] [ 7917.480204] Key type lgssc registered [ 7918.614965] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7926.341285] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7926.379342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7927.872677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7931.467782] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 7938.017647] LustreError: 232742:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7938.042319] LustreError: 232742:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7940.603698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7940.992928] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7945.833843] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 7953.780423] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7959.152674] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7965.628335] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7966.001205] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7972.452062] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_default_debug -1 all [ 7973.226586] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7980.257726] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7997.528311] Lustre: server umount lustre-OST0000 complete [ 8002.533026] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8002.540419] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8002.568503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8004.071212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8004.072774] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8009.187101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8009.197407] Lustre: Skipped 2 previous similar messages [ 8011.311620] Lustre: server umount lustre-MDT0000 complete [ 8014.304200] Lustre: 232055:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788450127/real 1788450127] req@0000000045ea8263 x1875325793373952/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788450143 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8014.310672] LustreError: 232738:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8014.330424] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8014.371857] LustreError: 232738:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8015.433541] LustreError: 232724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450144 with bad export cookie 17360647187058852084 [ 8015.435675] LustreError: MGC192.168.202.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8015.443415] LustreError: 232724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8015.871881] Lustre: server umount lustre-MDT0001 complete [ 8027.371872] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [ 8030.244949] Key type lgssc unregistered [ 8030.564525] LNet: 236433:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8030.572986] LNetError: 236433:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8030.588116] LNet: Removed LNI 192.168.202.157@tcp [ 8031.445268] Key type .llcrypt unregistered [ 8031.449817] Key type ._llcrypt unregistered [ 8054.262803] Key type ._llcrypt registered [ 8054.265631] Key type .llcrypt registered [ 8054.436984] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 8055.322220] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8055.380159] alg: No test for adler32 (adler32-zlib) [ 8056.663379] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 8057.060381] LNet: Added LNI 192.168.202.157@tcp [8/256/0/180] [ 8058.800685] Key type lgssc registered [ 8060.279580] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8333.975595] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [ 8336.988479] Key type lgssc unregistered [ 8337.347516] LNet: 238477:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8337.356603] LNetError: 238477:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8337.380285] LNet: Removed LNI 192.168.202.157@tcp [ 8338.363511] Key type .llcrypt unregistered [ 8338.367709] Key type ._llcrypt unregistered [ 8369.176887] Key type ._llcrypt registered [ 8369.178130] Key type .llcrypt registered [ 8369.225575] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing set_hostid [ 8388.913861] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing load_modules_local [ 8390.567450] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8390.663289] alg: No test for adler32 (adler32-zlib) [ 8391.692809] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 8391.892584] LNet: Added LNI 192.168.202.157@tcp [8/256/0/180] [ 8393.504197] Key type lgssc registered [ 8394.310124] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8425.065692] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 8087 sec ========== 11:49:13 (1788450553) [ 8426.811388] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 11:49:14 (1788450554) === [ 8430.501146] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 11:49:18 (1788450558) === [ 8471.373790] Lustre: DEBUG MARKER: oleg257-server.virtnet: executing unload_modules_local [ 8474.047974] Key type lgssc unregistered [ 8474.336599] LNet: 243716:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8474.352250] LNetError: 243716:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8474.377057] LNet: Removed LNI 192.168.202.157@tcp [ 8475.169366] Key type .llcrypt unregistered [ 8475.175046] Key type ._llcrypt unregistered