[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 476134970 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002436] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.005006] kvm-guest: setup PV IPIs [ 0.007725] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010013] pid_max: default: 32768 minimum: 301 [ 0.011136] LSM: Security Framework initializing [ 0.012060] Yama: becoming mindful. [ 0.013057] SELinux: Initializing. [ 0.014071] *** VALIDATE selinux *** [ 0.022818] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028637] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029144] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030136] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031111] *** VALIDATE tmpfs *** [ 0.032509] *** VALIDATE proc *** [ 0.033244] *** VALIDATE cgroup *** [ 0.034011] *** VALIDATE cgroup2 *** [ 0.036336] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037163] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039032] Spectre V2 : User space: Vulnerable [ 0.040012] Speculative Store Bypass: Vulnerable [ 0.043229] debug: unmapping init [mem 0xffffffff8fc59000-0xffffffff8fc60fff] [ 0.046132] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047693] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048025] ... version: 2 [ 0.049016] ... bit width: 48 [ 0.050015] ... generic registers: 4 [ 0.051010] ... value mask: 0000ffffffffffff [ 0.052011] ... max period: 00007fffffffffff [ 0.053015] ... fixed-purpose events: 3 [ 0.054015] ... event mask: 000000070000000f [ 0.056222] rcu: Hierarchical SRCU implementation. [ 0.058416] smp: Bringing up secondary CPUs ... [ 0.059657] x86: Booting SMP configuration: [ 0.060031] .... node #0, CPUs: #1 #2 #3 [ 0.064165] smp: Brought up 1 node, 4 CPUs [ 0.066018] smpboot: Max logical packages: 1 [ 0.067023] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.231745] node 0 deferred pages initialised in 162ms [ 0.236021] devtmpfs: initialized [ 0.237259] x86/mm: Memory block size: 128MB [ 0.239828] gcov: version magic: 0x41383552 [ 0.240616] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.245230] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.247493] pinctrl core: initialized pinctrl subsystem [ 0.250261] [ 0.250984] ************************************************************* [ 0.253018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.256085] ** ** [ 0.259020] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.261017] ** ** [ 0.264019] ** This means that this kernel is built to expose internal ** [ 0.267016] ** IOMMU data structures, which may compromise security on ** [ 0.269018] ** your system. ** [ 0.271020] ** ** [ 0.274088] ** If you see this message and you are not debugging the ** [ 0.277020] ** kernel, report this immediately to your vendor! ** [ 0.280018] ** ** [ 0.282020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.285018] ************************************************************* [ 0.288904] NET: Registered protocol family 16 [ 0.291532] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.294087] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.297085] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.301152] cpuidle: using governor menu [ 0.302899] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.306569] PCI: Using configuration type 1 for base access [ 0.308167] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.322018] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.323027] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.325183] cryptd: max_cpu_qlen set to 1000 [ 0.329189] ACPI: Added _OSI(Module Device) [ 0.331054] ACPI: Added _OSI(Processor Device) [ 0.333025] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.336027] ACPI: Added _OSI(Processor Aggregator Device) [ 0.343204] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.351229] ACPI: Interpreter enabled [ 0.352080] ACPI: PM: (supports S0 S3 S4 S5) [ 0.354027] ACPI: Using IOAPIC for interrupt routing [ 0.356144] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.360426] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.371096] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.373068] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.377030] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.380107] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.385422] acpiphp: Slot [2] registered [ 0.387151] acpiphp: Slot [5] registered [ 0.389184] acpiphp: Slot [6] registered [ 0.391151] acpiphp: Slot [7] registered [ 0.392124] acpiphp: Slot [8] registered [ 0.394257] acpiphp: Slot [9] registered [ 0.396144] acpiphp: Slot [10] registered [ 0.397147] acpiphp: Slot [3] registered [ 0.399129] acpiphp: Slot [4] registered [ 0.401142] acpiphp: Slot [11] registered [ 0.402174] acpiphp: Slot [12] registered [ 0.404146] acpiphp: Slot [13] registered [ 0.406136] acpiphp: Slot [14] registered [ 0.408155] acpiphp: Slot [15] registered [ 0.409133] acpiphp: Slot [16] registered [ 0.411145] acpiphp: Slot [17] registered [ 0.413179] acpiphp: Slot [18] registered [ 0.414135] acpiphp: Slot [19] registered [ 0.416166] acpiphp: Slot [20] registered [ 0.417109] acpiphp: Slot [21] registered [ 0.419111] acpiphp: Slot [22] registered [ 0.420158] acpiphp: Slot [23] registered [ 0.422160] acpiphp: Slot [24] registered [ 0.425060] acpiphp: Slot [25] registered [ 0.426146] acpiphp: Slot [26] registered [ 0.428230] acpiphp: Slot [27] registered [ 0.430147] acpiphp: Slot [28] registered [ 0.433184] acpiphp: Slot [29] registered [ 0.434145] acpiphp: Slot [30] registered [ 0.437139] acpiphp: Slot [31] registered [ 0.438073] PCI host bridge to bus 0000:00 [ 0.440024] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.444026] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.446025] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.450025] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.454045] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.457042] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.459203] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.461969] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.465227] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.475679] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.481036] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.484020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.487026] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.491020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.495470] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.498812] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.501054] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.503784] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.510015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.523014] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.530018] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.535793] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.544018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.552016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.569018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.581942] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.590019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.596018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.614022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.623921] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.632019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.637021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.652022] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.661244] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.666019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.673022] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.690018] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.703265] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.708017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.714021] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.728021] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.737821] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.744022] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.752021] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.773022] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.784880] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.787535] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.789412] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.793464] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.795346] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.801161] iommu: Default domain type: Passthrough [ 0.803605] SCSI subsystem initialized [ 0.805184] ACPI: bus type USB registered [ 0.807161] usbcore: registered new interface driver usbfs [ 0.809133] usbcore: registered new interface driver hub [ 0.811088] usbcore: registered new device driver usb [ 0.813233] pps_core: LinuxPPS API ver. 1 registered [ 0.814011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.816090] PTP clock support registered [ 0.819163] EDAC MC: Ver: 3.0.0 [ 0.821561] PCI: Using ACPI for IRQ routing [ 0.823986] NetLabel: Initializing [ 0.826017] NetLabel: domain hash size = 128 [ 0.827012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.829104] NetLabel: unlabeled traffic allowed by default [ 0.832052] vgaarb: loaded [ 0.833318] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.835020] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.842422] clocksource: Switched to clocksource kvm-clock [ 0.947891] VFS: Disk quotas dquot_6.6.0 [ 0.949784] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.952328] *** VALIDATE ramfs *** [ 0.953367] *** VALIDATE hugetlbfs *** [ 0.954804] pnp: PnP ACPI init [ 0.957507] pnp: PnP ACPI: found 6 devices [ 0.974797] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.978983] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.981710] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.985629] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.988509] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.990810] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.993519] NET: Registered protocol family 2 [ 0.995811] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.000741] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.003977] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.008742] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.012235] TCP: Hash tables configured (established 65536 bind 65536) [ 1.014966] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.017975] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.020647] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.023658] NET: Registered protocol family 1 [ 1.026136] RPC: Registered named UNIX socket transport module. [ 1.028139] RPC: Registered udp transport module. [ 1.031090] RPC: Registered tcp transport module. [ 1.033770] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.036463] NET: Registered protocol family 44 [ 1.038470] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.042586] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.045664] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.048743] PCI: CLS 0 bytes, default 64 [ 1.051376] Unpacking initramfs... [ 2.433408] debug: unmapping init [mem 0xffff892d7cc54000-0xffff892d7ffbffff] [ 2.437058] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.439958] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.443973] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.945185] Initialise system trusted keyrings [ 2.947167] Key type blacklist registered [ 2.949509] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.959938] zbud: loaded [ 2.963469] *** VALIDATE nfs *** [ 2.964959] *** VALIDATE nfs4 *** [ 2.966744] pstore: using deflate compression [ 2.970438] Platform Keyring initialized [ 3.072323] NET: Registered protocol family 38 [ 3.074129] Key type asymmetric registered [ 3.075721] Asymmetric key parser 'x509' registered [ 3.077106] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.080851] io scheduler mq-deadline registered [ 3.083422] io scheduler kyber registered [ 3.085917] io scheduler bfq registered [ 3.088149] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.090851] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.095318] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.098818] ACPI: Power Button [PWRF] [ 3.105140] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.112234] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.125651] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.131797] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.144753] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.172625] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.201202] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.208688] Non-volatile memory driver v1.3 [ 3.210453] Linux agpgart interface v0.103 [ 3.242798] virtio_blk virtio1: [vda] 149760 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.246255] vda: detected capacity change from 0 to 76677120 [ 3.262911] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.266373] vdb: detected capacity change from 0 to 1073741824 [ 3.285574] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.288875] vdc: detected capacity change from 0 to 2621440000 [ 3.307265] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.310980] vdd: detected capacity change from 0 to 2621440000 [ 3.328533] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.331121] vde: detected capacity change from 0 to 4294967296 [ 3.346489] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.349484] vdf: detected capacity change from 0 to 4294967296 [ 3.356815] libphy: Fixed MDIO Bus: probed [ 3.362252] usbcore: registered new interface driver usbserial_generic [ 3.365243] usbserial: USB Serial support registered for generic [ 3.368084] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.372957] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.375122] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.378032] mousedev: PS/2 mouse device common for all mice [ 3.381095] rtc_cmos 00:05: RTC can wake from S4 [ 3.384180] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.384966] rtc_cmos 00:05: registered as rtc0 [ 3.389967] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.390972] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.393771] intel_pstate: CPU model not supported [ 3.400557] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.401111] hid: raw HID events driver (C) Jiri Kosina [ 3.406308] usbcore: registered new interface driver usbhid [ 3.407952] usbhid: USB HID core driver [ 3.409449] drop_monitor: Initializing network drop monitor service [ 3.411961] Initializing XFRM netlink socket [ 3.414368] NET: Registered protocol family 10 [ 3.417805] Segment Routing with IPv6 [ 3.419256] NET: Registered protocol family 17 [ 3.421553] mpls_gso: MPLS GSO support [ 3.426480] RAS: Correctable Errors collector initialized. [ 3.428694] AVX version of gcm_enc/dec engaged. [ 3.430358] AES CTR mode by8 optimization enabled [ 3.512616] sched_clock: Marking stable (3512592414, 0)->(4448875202, -936282788) [ 3.516495] registered taskstats version 1 [ 3.519184] Loading compiled-in X.509 certificates [ 3.521160] zswap: loaded using pool lzo/zbud [ 3.548470] Key type big_key registered [ 3.561399] Key type encrypted registered [ 3.563283] ima: No TPM chip found, activating TPM-bypass! [ 3.565794] ima: Allocated hash algorithm: sha1 [ 3.567807] ima: No architecture policies found [ 3.569667] evm: Initialising EVM extended attributes: [ 3.571478] evm: security.selinux [ 3.572614] evm: security.ima [ 3.573895] evm: security.capability [ 3.575678] evm: HMAC attrs: 0x1 [ 3.578677] rtc_cmos 00:05: setting system clock to 2026-09-03 11:56:19 UTC (1788436579) [ 3.593269] debug: unmapping init [mem 0xffffffff90c03000-0xffffffff90dfffff] [ 3.596629] debug: unmapping init [mem 0xffffffff8f982000-0xffffffff8fc58fff] [ 3.605133] Write protecting the kernel read-only data: 28672k [ 3.608873] debug: unmapping init [mem 0xffffffff8e003000-0xffffffff8e1fffff] [ 3.612075] debug: unmapping init [mem 0xffffffff8e914000-0xffffffff8e9fffff] [ 3.669195] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.679278] systemd[1]: Detected virtualization kvm. [ 3.681396] systemd[1]: Detected architecture x86-64. [ 3.683372] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.708955] systemd[1]: No hostname configured. [ 3.711022] systemd[1]: Set hostname to . [ 3.712875] random: systemd: uninitialized urandom read (16 bytes read) [ 3.715312] systemd[1]: Initializing machine ID from random generator. [ 3.771440] random: ln: uninitialized urandom read (6 bytes read) [ 3.856192] random: systemd: uninitialized urandom read (16 bytes read) [ 3.859652] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.865167] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.870859] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Sockets. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Reached target Swap. Starting Apply Kernel Variables... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.571986] device-mapper: uevent: version 1.0.3 [ 4.574671] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.325301] random: fast init done [ 5.331984] virtio_net virtio0 ens2: renamed from eth0 [ 5.393284] scsi host0: ata_piix [ 5.411585] scsi host1: ata_piix [ 5.413501] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.416545] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.248987] dracut-initqueue[580]: RTNETLINK answers: File exists [ 10.084519] random: crng init done [ 10.086071] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.628811] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.851082] printk: systemd: 25 output lines suppressed due to ratelimiting [ 12.135794] SELinux: Disabled at runtime. [ 12.200076] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.212576] systemd[1]: Detected virtualization kvm. [ 12.215391] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.762267] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.765812] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.772078] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.780388] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.784117] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.793929] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.798642] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target rpc_pipefs.target. Mounting POSIX Message Queue File System... [ OK ] Created slice system-getty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. Starting Remount Root a[ 12.845391] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS nd Kernel File Systems... Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Mounting Huge Pages File System... [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Initrd File Systems. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.312468] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.715880] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.724050] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.912241] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.942675] EDAC sbridge: Ver: 1.1.2 [ 15.451876] Key type dns_resolver registered [ 15.778281] NFS: Registering the id_resolver key type [ 15.781109] Key type id_resolver registered [ 15.783227] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started RPC Bind. [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started dnf makecache --timer. [ OK ] Started D-Bus System Message Bus. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg357-server login: [ 50.174033] libcfs: loading out-of-tree module taints kernel. [ 50.230198] Key type ._llcrypt registered [ 50.232854] Key type .llcrypt registered [ 50.353350] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_hostid [ 67.995233] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 69.787197] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 69.801662] alg: No test for adler32 (adler32-zlib) [ 71.238687] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 71.932401] hrtimer: interrupt took 2608631 ns [ 72.099529] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 74.025779] Key type lgssc registered [ 76.417984] Lustre: Echo OBD driver; http://www.lustre.org/ [ 97.562843] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 142.543289] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 158.299919] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 158.356502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 159.857917] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 159.905274] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 160.101324] Lustre: lustre-MDT0000: new disk, initializing [ 160.317229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 160.360864] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 165.194299] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 179.130398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 179.196781] Lustre: 6521:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 179.219716] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 179.223644] Lustre: Skipped 1 previous similar message [ 179.287557] Lustre: lustre-MDT0001: new disk, initializing [ 179.336352] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 179.365454] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 179.375372] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 183.287547] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 187.982580] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 197.882096] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 198.136386] Lustre: lustre-OST0000: new disk, initializing [ 198.139737] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 198.145136] Lustre: 8459:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 198.213373] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 204.879993] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 204.890051] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 205.030596] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 205.269590] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 221.329618] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 221.537699] Lustre: lustre-OST0001: new disk, initializing [ 221.554424] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 221.565560] Lustre: 9533:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 221.670341] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 228.977353] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 231.514819] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 231.529793] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 231.611561] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 241.256980] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 248.202385] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 255.704664] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing check_logdir /tmp/testlogs/ [ 261.670223] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing yml_node [ 266.698609] Lustre: DEBUG MARKER: Client: 2.17.58.2 [ 269.220746] Lustre: DEBUG MARKER: MDS: 2.17.58.2 [ 272.113238] Lustre: DEBUG MARKER: OSS: 2.17.58.2 [ 273.759437] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 3 08:00:46 EDT 2026 [ 293.274990] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 295.076554] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 312.805950] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 312.819052] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 312.857413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 313.828376] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 313.829806] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 313.877540] Lustre: Skipped 1 previous similar message [ 319.125461] Lustre: server umount lustre-MDT0000 complete [ 323.600783] LustreError: 10349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788436899 with bad export cookie 11360997653627708686 [ 323.602433] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 323.621147] LustreError: 10349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 324.064652] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 324.067252] LustreError: 6530:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 324.076477] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 324.089851] Lustre: Skipped 3 previous similar messages [ 324.134387] LustreError: 6530:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 328.098828] LustreError: 8467:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 328.102863] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 328.159099] Lustre: Skipped 1 previous similar message [ 328.180931] LustreError: 8467:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 330.410563] Lustre: server umount lustre-MDT0001 complete [ 335.408362] Lustre: server umount lustre-OST0000 complete [ 340.097915] Lustre: server umount lustre-OST0001 complete [ 347.780725] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_hostid [ 358.112970] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 397.872701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 398.201868] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 398.230536] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 398.308540] Lustre: lustre-MDT0000: new disk, initializing [ 398.436396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 398.471147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 403.062825] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 414.073562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 414.241359] Lustre: 15101:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 414.340702] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 414.349484] Lustre: Skipped 1 previous similar message [ 414.495307] Lustre: lustre-MDT0001: new disk, initializing [ 414.598569] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 414.640555] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 414.656523] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 419.384663] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 429.247708] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 435.897376] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 443.108612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 443.497915] Lustre: lustre-OST0000: new disk, initializing [ 443.502507] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 443.510253] Lustre: 17070:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 443.620564] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 445.111160] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 445.136506] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 445.324091] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 451.273942] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 462.204897] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 467.498913] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 467.767428] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 472.185841] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 472.493263] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 476.130996] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 476.132400] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 476.155665] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 476.168831] Lustre: Skipped 2 previous similar messages [ 480.855654] Lustre: server umount lustre-OST0000 complete [ 486.368770] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 486.373406] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 486.394241] Lustre: Skipped 1 previous similar message [ 486.397181] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 486.430058] Lustre: Skipped 1 previous similar message [ 491.243037] Lustre: server umount lustre-MDT0000 complete [ 491.490116] LustreError: 16258:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 491.509369] LustreError: 16258:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 495.081522] LustreError: 15093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788437071 with bad export cookie 11360997653627710219 [ 495.085714] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 495.099907] LustreError: 15093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 495.479337] Lustre: server umount lustre-MDT0001 complete [ 499.006316] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 08:04:31 (1788437071) [ 505.879177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 506.530829] LustreError: 19335:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 506.606711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 511.292363] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 511.972682] LustreError: 19336:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 520.043851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 525.307939] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 533.554396] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 539.906876] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 546.162448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 556.026653] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 566.440596] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 573.408909] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 573.417445] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 573.430789] Lustre: Skipped 1 previous similar message [ 573.434870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 587.231408] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 587.430883] Lustre: server umount lustre-OST0000 complete [ 591.844988] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 591.859294] Lustre: Skipped 1 previous similar message [ 598.049699] Lustre: server umount lustre-MDT0000 complete [ 602.091478] LustreError: 19335:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 602.126948] LustreError: 19335:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 603.117854] LustreError: 19318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788437179 with bad export cookie 11360997653627711101 [ 603.120111] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 603.151019] LustreError: 19318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 603.602881] Lustre: server umount lustre-MDT0001 complete [ 615.922246] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 618.688079] Key type lgssc unregistered [ 618.956793] LNet: 22869:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 618.965604] LNetError: 22869:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 619.012161] LNet: Removed LNI 192.168.203.157@tcp [ 619.976306] Key type .llcrypt unregistered [ 619.980757] Key type ._llcrypt unregistered [ 645.251551] Key type ._llcrypt registered [ 645.254603] Key type .llcrypt registered [ 645.375914] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 646.444455] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 646.543472] alg: No test for adler32 (adler32-zlib) [ 647.672577] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 647.926792] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 649.607159] Key type lgssc registered [ 650.870579] Lustre: Echo OBD driver; http://www.lustre.org/ [ 659.468538] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 659.495687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 661.159621] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 665.746745] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 671.730607] LustreError: 24288:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 671.742827] LustreError: 24288:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 674.400378] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 674.716833] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 679.667385] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 687.203851] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 693.154103] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 699.975631] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 700.417337] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 706.546193] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 706.655739] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 713.339464] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 732.130502] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 732.136552] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 732.168793] Lustre: Skipped 1 previous similar message [ 732.171968] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 732.193987] Lustre: Skipped 1 previous similar message [ 733.607791] Lustre: server umount lustre-OST0000 complete [ 741.344912] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 741.357058] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 741.376148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 742.369118] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 742.381687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 742.386709] Lustre: Skipped 1 previous similar message [ 743.854977] Lustre: server umount lustre-MDT0000 complete [ 747.346073] LustreError: 24269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788437323 with bad export cookie 5652544257915161975 [ 747.356868] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 747.361229] LustreError: 24269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 747.905921] Lustre: server umount lustre-MDT0001 complete [ 758.508882] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 761.626547] Key type lgssc unregistered [ 761.865212] LNet: 28261:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 761.884903] LNetError: 28261:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 762.919964] LNet: Removed LNI 192.168.203.157@tcp [ 763.899800] Key type .llcrypt unregistered [ 763.903967] Key type ._llcrypt unregistered [ 788.590615] Key type ._llcrypt registered [ 788.596644] Key type .llcrypt registered [ 788.751480] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 789.865615] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 789.934113] alg: No test for adler32 (adler32-zlib) [ 791.125085] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 791.449777] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 793.247317] Key type lgssc registered [ 794.373586] Lustre: Echo OBD driver; http://www.lustre.org/ [ 801.789154] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 801.846799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 803.301816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 808.089569] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 813.548538] LustreError: 29675:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 813.559711] LustreError: 29675:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 816.519720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 816.987550] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 821.444701] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 828.923606] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 834.926030] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 848.378642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 848.699784] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 849.755558] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 855.031318] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 862.448948] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 870.368595] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 870.387825] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 870.413060] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 872.417505] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 872.417818] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 872.441056] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 874.715710] Lustre: server umount lustre-OST0000 complete [ 878.571481] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 878.587836] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 878.612859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 878.617464] Lustre: Skipped 1 previous similar message [ 882.660859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 882.661579] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 884.926583] Lustre: server umount lustre-MDT0000 complete [ 887.795623] LustreError: 29675:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 887.810362] LustreError: 29675:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 888.936497] LustreError: 32512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788437464 with bad export cookie 14346201249093650248 [ 888.953938] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 888.955018] LustreError: 32512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 889.394920] Lustre: server umount lustre-MDT0001 complete [ 902.419503] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 905.320087] Key type lgssc unregistered [ 905.600395] LNet: 33552:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 905.610856] LNetError: 33552:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 905.636949] LNet: Removed LNI 192.168.203.157@tcp [ 906.759875] Key type .llcrypt unregistered [ 906.764990] Key type ._llcrypt unregistered [ 934.463730] Key type ._llcrypt registered [ 934.470659] Key type .llcrypt registered [ 934.645646] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 936.129179] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 936.180458] alg: No test for adler32 (adler32-zlib) [ 937.115417] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 937.400519] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 939.185205] Key type lgssc registered [ 940.862100] Lustre: Echo OBD driver; http://www.lustre.org/ [ 949.327568] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 949.354051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 950.806426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 955.139723] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 960.994950] LustreError: 34963:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 961.009040] LustreError: 34963:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 962.663682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 962.997878] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 967.392436] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 974.672953] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 980.224279] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 987.003912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 987.368774] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 993.344500] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 995.566559] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1000.436818] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1009.130636] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1009.143077] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1009.158663] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1010.166873] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 1010.182897] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1010.199503] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1014.461130] Lustre: server umount lustre-OST0000 complete [ 1019.360416] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1019.365643] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1019.400317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1019.405700] Lustre: Skipped 2 previous similar messages [ 1021.409604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1021.416129] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1024.320931] Lustre: server umount lustre-MDT0000 complete [ 1026.537776] LustreError: 34963:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1028.377324] LustreError: 34942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788437604 with bad export cookie 10203428524912846335 [ 1028.390754] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1028.392243] LustreError: 34942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1028.779797] Lustre: server umount lustre-MDT0001 complete [ 1040.192244] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 1043.040770] Key type lgssc unregistered [ 1043.459166] LNet: 38494:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1043.467919] LNetError: 38494:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1044.522791] LNet: Removed LNI 192.168.203.157@tcp [ 1045.547070] Key type .llcrypt unregistered [ 1045.551671] Key type ._llcrypt unregistered [ 1070.391760] Key type ._llcrypt registered [ 1070.393552] Key type .llcrypt registered [ 1070.507896] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 1071.622487] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1071.799058] alg: No test for adler32 (adler32-zlib) [ 1072.900191] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1073.108509] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 1074.791271] Key type lgssc registered [ 1076.087967] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1084.072474] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1084.094675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1085.605257] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1089.804653] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1095.137608] LustreError: 39901:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1095.147777] LustreError: 39901:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1098.029625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1098.471614] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1102.730650] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1109.924399] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1115.316293] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1121.906066] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1122.251483] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1128.886647] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1130.498580] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1136.186788] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1143.781720] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1143.792743] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1143.811372] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1145.827157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1145.830993] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1145.835799] Lustre: Skipped 1 previous similar message [ 1148.447894] Lustre: server umount lustre-OST0000 complete [ 1170.284511] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1170.303571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1170.308880] Lustre: Skipped 2 previous similar messages [ 1174.654303] Lustre: server umount lustre-MDT0000 complete [ 1175.524943] LustreError: 40631:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1175.559438] LustreError: 40631:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1178.335726] LustreError: 40638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788437754 with bad export cookie 2554632362514767079 [ 1178.342156] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1178.362972] LustreError: 40638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1184.937255] Lustre: server umount lustre-MDT0001 complete [ 1197.108248] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 1200.117601] Key type lgssc unregistered [ 1200.419598] LNet: 43610:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1200.428926] LNetError: 43610:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1201.447030] LNet: Removed LNI 192.168.203.157@tcp [ 1202.405470] Key type .llcrypt unregistered [ 1202.407270] Key type ._llcrypt unregistered [ 1226.518869] Key type ._llcrypt registered [ 1226.521646] Key type .llcrypt registered [ 1226.642551] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 1227.521500] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1227.637085] alg: No test for adler32 (adler32-zlib) [ 1228.719473] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1229.001651] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 1230.735152] Key type lgssc registered [ 1232.033553] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1240.329304] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1240.350777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1241.858141] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1247.903664] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1251.302232] LustreError: 45019:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1251.339443] LustreError: 45019:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1256.420279] LustreError: 45020:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1257.474570] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1257.776166] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1262.313939] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1270.362694] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1276.099511] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1283.056146] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1283.435645] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1286.515935] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1289.430581] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1298.100391] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1309.156255] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1309.166129] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1309.189670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1311.434163] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.57@tcp (stopping) [ 1313.274495] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1313.278481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1313.291492] Lustre: Skipped 1 previous similar message [ 1313.313094] Lustre: Skipped 2 previous similar messages [ 1315.261586] Lustre: server umount lustre-MDT0000 complete [ 1316.527067] LustreError: 45015:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1319.490686] LustreError: 45000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788437895 with bad export cookie 621113775874804219 [ 1319.509164] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1321.657301] LustreError: 46034:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1321.677581] LustreError: 46034:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1321.697956] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.57@tcp (stopping) [ 1323.492120] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1326.061259] Lustre: server umount lustre-MDT0001 complete [ 1342.196934] Lustre: server umount lustre-OST0000 complete [ 1361.985192] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 1365.036175] Key type lgssc unregistered [ 1365.304784] LNet: 48881:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1365.314490] LNetError: 48881:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1365.345523] LNet: Removed LNI 192.168.203.157@tcp [ 1366.595069] Key type .llcrypt unregistered [ 1366.599120] Key type ._llcrypt unregistered [ 1393.929820] Key type ._llcrypt registered [ 1393.932125] Key type .llcrypt registered [ 1394.023313] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 1395.301939] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1395.476226] alg: No test for adler32 (adler32-zlib) [ 1396.719277] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1397.032150] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 1398.791230] Key type lgssc registered [ 1400.292679] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1408.806981] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1408.841558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1424.479658] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1429.600858] LustreError: 50267:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.157@tcp: failed processing log, type 4: rc = -110 [ 1456.223289] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1463.317418] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1471.363583] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1635.372681] Lustre: server umount lustre-OST0000 complete [ 1649.230422] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 1651.986552] Key type lgssc unregistered [ 1652.252474] LNet: 51939:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1652.268520] LNetError: 51939:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1652.288437] LNet: Removed LNI 192.168.203.157@tcp [ 1653.268464] Key type .llcrypt unregistered [ 1653.270303] Key type ._llcrypt unregistered [ 1677.199989] Key type ._llcrypt registered [ 1677.202992] Key type .llcrypt registered [ 1677.314766] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 1678.568673] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1678.588179] alg: No test for adler32 (adler32-zlib) [ 1679.625541] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1679.919696] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 1681.607179] Key type lgssc registered [ 1682.776152] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1691.207899] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1691.237847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1692.895311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1697.848299] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1702.374493] LustreError: 53350:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1702.396645] LustreError: 53350:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1706.207607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1706.694484] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1711.261721] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1717.982260] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1723.323656] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1729.855434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1730.194662] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1736.845836] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1737.395962] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1744.135436] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1746.912944] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1746.918489] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1746.937471] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1747.946235] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1747.948109] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1747.949116] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1747.962958] Lustre: Skipped 1 previous similar message [ 1753.060477] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1753.064738] Lustre: Skipped 1 previous similar message [ 1758.187044] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1758.193039] Lustre: Skipped 1 previous similar message [ 1761.247321] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1761.456801] Lustre: server umount lustre-OST0000 complete [ 1768.417572] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1768.426400] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1768.437977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1768.444884] Lustre: Skipped 1 previous similar message [ 1768.464815] Lustre: Skipped 1 previous similar message [ 1771.436200] Lustre: server umount lustre-MDT0000 complete [ 1773.536984] LustreError: 53350:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1773.555402] LustreError: 53350:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1775.105126] LustreError: 54085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438351 with bad export cookie 10985488899898701746 [ 1775.118114] LustreError: 54085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1775.119748] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1775.505099] Lustre: server umount lustre-MDT0001 complete [ 1787.243553] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 1790.202488] Key type lgssc unregistered [ 1790.530270] LNet: 56883:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1790.539468] LNetError: 56883:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1790.569725] LNet: Removed LNI 192.168.203.157@tcp [ 1791.378112] Key type .llcrypt unregistered [ 1791.387383] Key type ._llcrypt unregistered [ 1816.095385] Key type ._llcrypt registered [ 1816.100743] Key type .llcrypt registered [ 1816.229028] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 1817.444317] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1817.591865] alg: No test for adler32 (adler32-zlib) [ 1818.665069] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1819.031266] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 1820.695200] Key type lgssc registered [ 1821.912555] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1830.296470] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1830.324568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1845.919446] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1851.039343] LustreError: 58281:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.157@tcp: failed processing log, type 4: rc = -110 [ 1877.729374] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1884.303041] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1892.520524] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1899.453662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1900.106553] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1900.199865] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1904.766971] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1905.137436] LustreError: 58306:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1905.159123] LustreError: 58306:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1910.243255] LustreError: 58307:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1912.307137] LustreError: 59318:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1912.323133] LustreError: 59318:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1913.677705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1914.212142] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1918.786104] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1925.699556] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1932.137763] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1934.816650] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1934.823675] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1934.828055] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1934.851489] Lustre: Skipped 1 previous similar message [ 1939.937877] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1939.949563] Lustre: Skipped 2 previous similar messages [ 1940.908074] Lustre: server umount lustre-OST0000 complete [ 1961.562612] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1961.583793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1961.586598] Lustre: Skipped 1 previous similar message [ 1964.643059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1964.653069] Lustre: Skipped 1 previous similar message [ 1968.039953] Lustre: server umount lustre-MDT0000 complete [ 1970.164219] LustreError: 59323:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1970.206064] LustreError: 59323:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1972.739887] LustreError: 58287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438548 with bad export cookie 8325704854934079648 [ 1972.752513] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1972.759453] LustreError: 58287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1979.216600] Lustre: server umount lustre-MDT0001 complete [ 1990.908649] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 1993.894383] Key type lgssc unregistered [ 1994.166730] LNet: 62024:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1994.179725] LNetError: 62024:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1994.210259] LNet: Removed LNI 192.168.203.157@tcp [ 1995.244935] Key type .llcrypt unregistered [ 1995.251934] Key type ._llcrypt unregistered [ 2017.259647] Key type ._llcrypt registered [ 2017.262075] Key type .llcrypt registered [ 2017.402470] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 2018.358041] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2018.575827] alg: No test for adler32 (adler32-zlib) [ 2019.816163] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2020.115530] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 2021.808224] Key type lgssc registered [ 2022.980063] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2030.370522] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2030.395903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2031.855204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2036.134615] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2041.313150] LustreError: 63430:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2041.335349] LustreError: 63430:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2045.123583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2045.478834] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2049.894280] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2057.016514] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2063.209623] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2071.627080] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2072.079742] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2076.140440] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2079.809554] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2088.694504] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2105.836558] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2105.859398] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2105.868628] Lustre: Skipped 1 previous similar message [ 2107.910591] Lustre: server umount lustre-OST0000 complete [ 2111.969294] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2111.987240] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2112.001445] Lustre: Skipped 1 previous similar message [ 2112.024819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2116.064388] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2116.064387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2116.095476] Lustre: Skipped 1 previous similar message [ 2117.813972] Lustre: server umount lustre-MDT0000 complete [ 2121.184684] LustreError: 65273:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2121.215227] LustreError: 65273:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2121.612353] LustreError: 63413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438697 with bad export cookie 7628483707297922213 [ 2121.625974] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2121.629769] LustreError: 63413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2121.970713] Lustre: server umount lustre-MDT0001 complete [ 2132.625888] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 2135.262458] Key type lgssc unregistered [ 2135.535572] LNet: 66988:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2135.540473] LNetError: 66988:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2135.563288] LNet: Removed LNI 192.168.203.157@tcp [ 2136.466679] Key type .llcrypt unregistered [ 2136.468465] Key type ._llcrypt unregistered [ 2162.738627] Key type ._llcrypt registered [ 2162.740917] Key type .llcrypt registered [ 2162.822196] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 2163.894644] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2163.961559] alg: No test for adler32 (adler32-zlib) [ 2165.065762] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2165.347770] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 2167.034229] Key type lgssc registered [ 2167.983697] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2176.798392] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 08:32:29 (1788438749) [ 2183.494907] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2183.508661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2185.008579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2190.229649] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2194.401989] LustreError: 68690:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2194.421806] LustreError: 68690:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2199.125497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2199.604509] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2204.575883] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2212.399630] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2218.133423] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2224.722646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2225.047825] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2229.186934] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2231.508474] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2239.058402] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2247.583982] Lustre: Failing over lustre-MDT0000 [ 2247.850381] Lustre: server umount lustre-MDT0000 complete [ 2249.703427] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2249.718659] LustreError: 70757:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2249.720395] Lustre: Skipped 2 previous similar messages [ 2249.738762] LustreError: 70757:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2250.957741] LustreError: 70757:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2254.818271] LustreError: 69675:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2254.845628] LustreError: 69675:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2256.781660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2256.916818] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2257.110962] Lustre: *** cfs_fail_loc=135, val=0*** [ 2257.212709] LustreError: 71778:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2257.217410] LustreError: 71778:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.157@tcp: cfg command failed: rc = -2 [ 2257.224411] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2257.224411] [ 2257.230925] LustreError: MGC192.168.203.157@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2257.239087] LustreError: 71766:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2257.244056] LustreError: 71766:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [ 2257.247871] LustreError: 71766:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 2257.251318] LustreError: 71766:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2257.283274] Lustre: server umount lustre-MDT0000 complete [ 2257.291105] LustreError: 71766:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2259.940841] LustreError: 68691:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2259.959804] LustreError: 68691:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2263.693795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2263.815383] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2263.828116] LustreError: 68682:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2263.837022] LustreError: 72123:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGC192.168.203.157@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (00000000f2cc1417) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 2264.199220] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2264.260232] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2266.286080] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2268.409712] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2269.677257] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2269.704355] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2269.741708] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2279.910940] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2279.911766] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2279.925873] Lustre: Skipped 1 previous similar message [ 2282.307069] Lustre: server umount lustre-OST0000 complete [ 2290.146622] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2290.147460] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2290.165679] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2290.178435] Lustre: Skipped 1 previous similar message [ 2290.198184] Lustre: Skipped 2 previous similar messages [ 2291.997695] Lustre: server umount lustre-MDT0000 complete [ 2295.264665] LustreError: 69418:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2295.290193] LustreError: 69418:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2295.665822] LustreError: 68671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438871 with bad export cookie 17179500658652290740 [ 2295.679128] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2296.132835] Lustre: server umount lustre-MDT0001 complete [ 2307.310839] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 2310.326442] Key type lgssc unregistered [ 2310.678140] LNet: 73554:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2310.690461] LNetError: 73554:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2310.718571] LNet: Removed LNI 192.168.203.157@tcp [ 2311.728481] Key type .llcrypt unregistered [ 2311.733051] Key type ._llcrypt unregistered [ 2337.107399] Key type ._llcrypt registered [ 2337.112759] Key type .llcrypt registered [ 2337.272263] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 2338.250815] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2338.403721] alg: No test for adler32 (adler32-zlib) [ 2339.548967] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2339.858099] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 2341.615356] Key type lgssc registered [ 2342.939791] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2350.558473] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2350.576808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2351.985992] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2356.708576] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2361.317661] LustreError: 74965:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2361.339040] LustreError: 74965:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2365.304282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2365.721826] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2369.832584] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2377.755528] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2383.761956] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2390.751336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2391.153960] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2396.266298] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2397.327720] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2405.065916] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2414.879855] Lustre: Failing over lustre-MDT0000 [ 2415.132936] Lustre: server umount lustre-MDT0000 complete [ 2416.106932] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2416.113031] LustreError: 77456:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2416.120080] Lustre: Skipped 1 previous similar message [ 2418.379304] LustreError: 74959:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2418.408137] LustreError: 74959:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2421.219501] LustreError: 74965:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2421.247702] LustreError: 74965:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2424.744134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2424.900653] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2425.073353] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2425.191976] LustreError: 78051:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2425.203172] LustreError: 78051:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.157@tcp: cfg command failed: rc = -9 [ 2425.228109] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2425.228109] [ 2425.239511] LustreError: MGC192.168.203.157@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2425.257812] LustreError: 78040:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2425.279805] LustreError: 78040:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -9 [ 2425.300198] LustreError: 78040:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 2425.305425] LustreError: 78040:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2425.346901] Lustre: server umount lustre-MDT0000 complete [ 2425.349051] LustreError: 78040:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2426.336860] LustreError: 74964:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2426.379538] LustreError: 74964:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2432.741560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2432.946054] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2432.972346] LustreError: 74956:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2433.330134] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2433.389123] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2433.719815] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2437.812978] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2438.644986] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2438.710780] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2438.779774] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2446.816319] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2446.827053] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2446.841245] Lustre: Skipped 1 previous similar message [ 2446.857382] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2448.878141] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2448.880695] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2448.905229] Lustre: Skipped 1 previous similar message [ 2452.919560] Lustre: server umount lustre-OST0000 complete [ 2459.104318] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2459.115598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2459.119383] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2459.121204] Lustre: Skipped 1 previous similar message [ 2462.929180] Lustre: server umount lustre-MDT0000 complete [ 2464.227128] LustreError: 75692:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2464.242990] LustreError: 75692:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2466.732641] LustreError: 74945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439042 with bad export cookie 3848804606242353701 [ 2466.741234] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2466.752835] LustreError: 74945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2467.270775] Lustre: server umount lustre-MDT0001 complete [ 2477.889067] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 2480.873509] Key type lgssc unregistered [ 2481.162986] LNet: 79834:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2481.168660] LNetError: 79834:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2481.182627] LNet: Removed LNI 192.168.203.157@tcp [ 2482.139160] Key type .llcrypt unregistered [ 2482.144081] Key type ._llcrypt unregistered [ 2494.049563] Key type ._llcrypt registered [ 2494.051732] Key type .llcrypt registered [ 2494.167262] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 2494.479275] Key type .llcrypt unregistered [ 2494.483591] Key type ._llcrypt unregistered [ 2517.544738] Key type ._llcrypt registered [ 2517.552248] Key type .llcrypt registered [ 2517.702170] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 2518.593001] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2518.610294] alg: No test for adler32 (adler32-zlib) [ 2519.859891] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2520.227742] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 2521.943939] Key type lgssc registered [ 2523.055552] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2530.242716] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2530.266981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2531.920227] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2536.572138] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2542.054873] LustreError: 81729:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2542.073364] LustreError: 81729:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2544.963777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2545.278343] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2548.855380] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2555.763057] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2561.649870] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2568.578961] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2569.043203] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2575.293215] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2576.318575] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2582.013867] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2590.688352] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2590.698206] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2590.722032] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2591.716263] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2591.718336] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2591.720706] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2591.764128] Lustre: Skipped 1 previous similar message [ 2596.268932] Lustre: server umount lustre-OST0000 complete [ 2601.443698] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2601.457647] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2601.482591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2605.856181] Lustre: server umount lustre-MDT0000 complete [ 2607.074599] LustreError: 81729:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2607.090609] LustreError: 81729:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2609.354509] LustreError: 81709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439185 with bad export cookie 5152550564043713113 [ 2609.355931] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2609.360611] LustreError: 81709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2609.620690] Lustre: server umount lustre-MDT0001 complete [ 2621.110463] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 2624.030103] Key type lgssc unregistered [ 2624.567212] LNet: 85260:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2624.583063] LNetError: 85260:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2624.636454] LNet: Removed LNI 192.168.203.157@tcp [ 2625.951842] Key type .llcrypt unregistered [ 2625.953545] Key type ._llcrypt unregistered [ 2646.267054] Key type ._llcrypt registered [ 2646.268189] Key type .llcrypt registered [ 2646.353359] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 2647.328182] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2647.361254] alg: No test for adler32 (adler32-zlib) [ 2648.448713] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2648.695103] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 2650.423167] Key type lgssc registered [ 2651.696710] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2660.497758] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2660.552883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2662.100081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2667.220546] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2671.592131] LustreError: 86668:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2671.624535] LustreError: 86668:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2676.705957] LustreError: 86669:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2677.056857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2677.374319] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2682.445459] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2690.680724] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2696.785924] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2704.606772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2704.911656] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2707.002083] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2712.364568] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2720.089091] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2732.517068] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2732.520605] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2732.523712] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2732.534788] Lustre: Skipped 1 previous similar message [ 2732.541110] Lustre: Skipped 1 previous similar message [ 2737.635734] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2737.657357] Lustre: Skipped 1 previous similar message [ 2742.753375] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2742.763150] Lustre: Skipped 1 previous similar message [ 2744.287122] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2744.572320] Lustre: server umount lustre-OST0000 complete [ 2748.899743] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2748.909774] LustreError: Skipped 1 previous similar message [ 2748.913551] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2748.925956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2752.995603] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2752.997539] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2754.808732] Lustre: server umount lustre-MDT0000 complete [ 2758.113745] LustreError: 86669:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2758.132803] LustreError: 86669:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2759.302367] LustreError: 86650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439335 with bad export cookie 10236637890615291231 [ 2759.312266] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2759.322421] LustreError: 86650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2759.622768] Lustre: server umount lustre-MDT0001 complete [ 2770.902267] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 2773.659226] Key type lgssc unregistered [ 2774.001783] LNet: 90201:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2774.017496] LNetError: 90201:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2775.082726] LNet: Removed LNI 192.168.203.157@tcp [ 2776.114971] Key type .llcrypt unregistered [ 2776.124285] Key type ._llcrypt unregistered [ 2798.365071] Key type ._llcrypt registered [ 2798.368176] Key type .llcrypt registered [ 2798.522182] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 2799.287481] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2799.319216] alg: No test for adler32 (adler32-zlib) [ 2800.364381] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2800.545291] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 2802.199184] Key type lgssc registered [ 2803.533528] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2812.235549] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2812.277831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2813.835448] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2818.136838] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2823.141324] LustreError: 91611:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2823.159431] LustreError: 91611:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2826.500908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2826.962625] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2831.670775] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2838.859990] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2844.351721] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2851.143830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2851.475690] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2858.095721] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2858.675769] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2865.279255] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2877.931295] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2877.951210] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2877.968794] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2878.433429] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2879.463620] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2879.466462] Lustre: Skipped 2 previous similar messages [ 2882.459728] Lustre: server umount lustre-OST0000 complete [ 2888.162786] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2888.172215] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2888.189921] Lustre: Skipped 1 previous similar message [ 2888.200739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2889.705023] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2892.973684] Lustre: server umount lustre-MDT0000 complete [ 2894.817387] LustreError: 92631:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2894.844213] LustreError: 92631:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2896.339745] LustreError: 91591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439472 with bad export cookie 6040466540425890348 [ 2896.344863] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2896.348717] LustreError: 91591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2896.606565] Lustre: server umount lustre-MDT0001 complete [ 2908.985672] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 2912.686616] Key type lgssc unregistered [ 2913.059842] LNet: 95145:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2913.065314] LNetError: 95145:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2913.082649] LNet: Removed LNI 192.168.203.157@tcp [ 2913.833821] Key type .llcrypt unregistered [ 2913.836992] Key type ._llcrypt unregistered [ 2939.206365] Key type ._llcrypt registered [ 2939.208193] Key type .llcrypt registered [ 2939.383167] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 2940.080626] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2940.126689] alg: No test for adler32 (adler32-zlib) [ 2941.248953] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2941.570465] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 2943.231146] Key type lgssc registered [ 2944.650972] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2954.738821] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2954.777106] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2970.463432] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2975.583337] LustreError: 96528:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.157@tcp: failed processing log, type 4: rc = -110 [ 3002.272505] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3008.942407] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3017.135735] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3023.161234] Lustre: server umount lustre-OST0000 complete [ 3031.986831] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 08:46:44 (1788439604) [ 3039.469610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3039.977233] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3044.871554] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3045.355394] LustreError: 98176:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3045.397291] LustreError: 98176:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3049.444554] LustreError: 98177:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3053.515947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3053.749550] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3058.740283] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3066.341686] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3071.998439] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3078.073996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3078.440032] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3084.623077] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3085.054811] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3092.834987] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3105.258880] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3105.268564] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3105.286422] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3105.297492] Lustre: Skipped 1 previous similar message [ 3110.244157] Lustre: server umount lustre-OST0000 complete [ 3115.493635] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3115.494098] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3115.520838] Lustre: Skipped 1 previous similar message [ 3115.524779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3115.529671] LustreError: Skipped 1 previous similar message [ 3115.548822] Lustre: Skipped 1 previous similar message [ 3120.614900] LustreError: 98176:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3120.627331] LustreError: 98176:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3120.766348] Lustre: server umount lustre-MDT0000 complete [ 3124.677895] LustreError: 98157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439700 with bad export cookie 6791156918090876795 [ 3124.680026] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3124.694334] LustreError: 98157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3125.191966] Lustre: server umount lustre-MDT0001 complete [ 3138.376931] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 3141.280659] Key type lgssc unregistered [ 3141.589720] LNet: 101865:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3141.594699] LNetError: 101865:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3141.615457] LNet: Removed LNI 192.168.203.157@tcp [ 3142.450087] Key type .llcrypt unregistered [ 3142.452843] Key type ._llcrypt unregistered [ 3184.397665] Key type ._llcrypt registered [ 3184.400208] Key type .llcrypt registered [ 3184.540675] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 3185.382617] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3185.419551] alg: No test for adler32 (adler32-zlib) [ 3186.621872] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3186.844226] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 3188.503216] Key type lgssc registered [ 3189.731253] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3197.332706] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3197.362861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3198.872631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3203.018973] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3208.160401] LustreError: 104048:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3208.175662] LustreError: 104048:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3211.107978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3211.439961] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3216.090919] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3224.507817] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3231.203778] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3238.449300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3238.792259] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3243.956836] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3245.865550] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3254.012445] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3264.482506] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3264.500145] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3264.510770] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3266.528968] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3266.538572] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3266.551675] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3266.580512] Lustre: Skipped 1 previous similar message [ 3271.650264] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3271.656582] Lustre: Skipped 1 previous similar message [ 3276.257376] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3276.488447] Lustre: server umount lustre-OST0000 complete [ 3281.890200] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3281.917455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3281.929031] Lustre: Skipped 1 previous similar message [ 3286.340935] Lustre: server umount lustre-MDT0000 complete [ 3287.010215] LustreError: 104048:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3287.021119] LustreError: 104048:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3290.337478] LustreError: 104784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439866 with bad export cookie 13025411458838613710 [ 3290.343527] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3290.347913] LustreError: 104784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3290.664106] Lustre: server umount lustre-MDT0001 complete [ 3303.418177] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 3306.422964] Key type lgssc unregistered [ 3306.725966] LNet: 107582:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3306.733460] LNetError: 107582:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3306.743700] LNet: Removed LNI 192.168.203.157@tcp [ 3307.660894] Key type .llcrypt unregistered [ 3307.663154] Key type ._llcrypt unregistered [ 3325.555616] Key type ._llcrypt registered [ 3325.560206] Key type .llcrypt registered [ 3325.684501] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 3326.947405] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3326.978187] alg: No test for adler32 (adler32-zlib) [ 3328.027232] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3328.341412] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 3330.047150] Key type lgssc registered [ 3331.271534] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3339.239547] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3339.259815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3354.783851] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3359.903526] LustreError: 108824:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.157@tcp: failed processing log, type 4: rc = -110 [ 3386.655193] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3395.814951] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3402.998795] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3410.530163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3410.764473] LustreError: MGC192.168.203.157@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3410.778928] LustreError: 109848:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3410.785514] LustreError: 109848:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [ 3410.797735] LustreError: 109848:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 3410.865670] Lustre: server umount lustre-MDT0000 complete [ 3410.872549] LustreError: 109848:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3430.504653] Lustre: server umount lustre-OST0000 complete [ 3440.156113] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_hostid [ 3449.683307] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 3485.867081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3486.090572] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3486.154075] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3486.260242] Lustre: lustre-MDT0000: new disk, initializing [ 3486.324188] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3486.347650] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3490.544742] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3500.617515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3500.752846] Lustre: 112642:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3500.840367] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3500.856495] Lustre: Skipped 1 previous similar message [ 3500.930569] Lustre: lustre-MDT0001: new disk, initializing [ 3501.033260] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3501.061879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3501.076050] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3505.583019] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3514.062709] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3519.770729] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3525.883553] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3526.269229] Lustre: lustre-OST0000: new disk, initializing [ 3526.273435] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3526.285780] Lustre: 114610:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3526.447698] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3532.580273] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3535.466566] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3535.479842] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3535.646159] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3541.703944] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3546.252168] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3546.539583] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3551.197900] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3551.365925] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3556.324499] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3556.325504] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3556.362633] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3559.673484] Lustre: server umount lustre-OST0000 complete [ 3566.561624] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3566.564788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3566.592498] Lustre: Skipped 2 previous similar messages [ 3566.618097] Lustre: Skipped 2 previous similar messages [ 3570.174297] Lustre: server umount lustre-MDT0000 complete [ 3571.687642] LustreError: 112648:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3571.709073] LustreError: 112648:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3574.548102] LustreError: 112633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440150 with bad export cookie 14252813939365676526 [ 3574.560833] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3574.569191] LustreError: 112633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3575.152231] Lustre: server umount lustre-MDT0001 complete [ 3583.879824] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 08:55:56 (1788440156) [ 3586.778300] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3603.722826] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_hostid [ 3613.655401] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 3649.364695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3649.630788] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3649.658061] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3649.755762] Lustre: lustre-MDT0000: new disk, initializing [ 3649.875741] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3649.898379] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3654.234953] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3664.956197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3665.062292] Lustre: 119509:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3665.153447] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3665.159428] Lustre: Skipped 1 previous similar message [ 3665.240364] Lustre: lustre-MDT0001: new disk, initializing [ 3665.315978] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3665.347264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3665.367345] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3670.211821] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3680.235217] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3685.940462] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3692.510371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3692.793141] Lustre: lustre-OST0000: new disk, initializing [ 3692.805200] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3692.830382] Lustre: 121478:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3692.934132] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3699.527100] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3701.854176] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3701.862961] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3701.963814] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3709.281392] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3714.454411] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3714.703943] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3720.259067] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3720.643282] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3722.721260] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3722.727518] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3722.731695] LustreError: Skipped 1 previous similar message [ 3722.734518] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3722.761350] Lustre: Skipped 1 previous similar message [ 3722.787724] Lustre: Skipped 1 previous similar message [ 3727.853817] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3727.868516] Lustre: Skipped 1 previous similar message [ 3728.985647] Lustre: server umount lustre-OST0000 complete [ 3732.963947] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3732.976536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3732.994787] Lustre: Skipped 1 previous similar message [ 3738.968819] Lustre: server umount lustre-MDT0000 complete [ 3742.589709] LustreError: 119504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440318 with bad export cookie 14252813939365677415 [ 3742.595646] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3742.606647] LustreError: 119504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3743.016736] Lustre: server umount lustre-MDT0001 complete [ 3752.202567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3752.873992] LustreError: 123693:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3752.961672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3758.050543] LustreError: 123694:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3758.398663] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3762.144849] LustreError: 123693:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3767.196759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3767.273848] LustreError: 123694:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3772.559987] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3780.122793] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3785.898612] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3792.772401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3799.465924] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3807.271451] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3818.247509] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3823.072258] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3823.075734] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3823.098847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3823.118046] Lustre: Skipped 1 previous similar message [ 3823.142792] Lustre: Skipped 4 previous similar messages [ 3828.927271] Lustre: server umount lustre-OST0000 complete [ 3833.312478] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3833.322719] Lustre: Skipped 1 previous similar message [ 3838.933468] Lustre: server umount lustre-MDT0000 complete [ 3842.658497] LustreError: 123673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440418 with bad export cookie 14252813939365678290 [ 3842.668091] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3842.677076] LustreError: 123673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3843.125691] Lustre: server umount lustre-MDT0001 complete [ 3854.883072] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 3858.367811] Key type lgssc unregistered [ 3858.649844] LNet: 127325:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3858.672507] LNetError: 127325:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3858.688678] LNet: Removed LNI 192.168.203.157@tcp [ 3859.579485] Key type .llcrypt unregistered [ 3859.585656] Key type ._llcrypt unregistered [ 3877.624892] Key type ._llcrypt registered [ 3877.626879] Key type .llcrypt registered [ 3877.741860] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_hostid [ 3894.252819] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 3895.663339] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3895.685683] alg: No test for adler32 (adler32-zlib) [ 3896.852513] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3897.129109] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 3898.839213] Key type lgssc registered [ 3900.191119] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3935.065521] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3935.085144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3936.445890] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3936.489997] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3936.654510] Lustre: lustre-MDT0000: new disk, initializing [ 3936.723083] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3936.750950] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3942.235925] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3955.621071] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3955.775742] Lustre: 130486:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3955.854284] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3955.858321] Lustre: Skipped 1 previous similar message [ 3955.979983] Lustre: lustre-MDT0001: new disk, initializing [ 3956.149027] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3956.192266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3956.205074] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3962.068139] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3971.824504] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3977.141872] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3983.375884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3983.739515] Lustre: lustre-OST0000: new disk, initializing [ 3983.745333] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3983.769796] Lustre: 132459:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3983.873828] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3990.090530] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3990.102793] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3990.231869] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3990.233466] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3999.566766] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4004.434574] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4004.699629] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4009.541692] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4009.838555] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4016.101630] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4016.112132] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4016.126733] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4021.222078] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4021.227788] Lustre: Skipped 2 previous similar messages [ 4026.335121] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4026.339590] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4026.358535] Lustre: Skipped 1 previous similar message [ 4026.585257] Lustre: server umount lustre-OST0000 complete [ 4031.456519] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4031.460476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4031.488046] Lustre: Skipped 2 previous similar messages [ 4036.578505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4036.589733] Lustre: Skipped 2 previous similar messages [ 4036.816916] Lustre: server umount lustre-MDT0000 complete [ 4040.718612] LustreError: 130479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440616 with bad export cookie 2668723956623327373 [ 4040.730645] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4040.733415] LustreError: 130479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4041.059149] Lustre: server umount lustre-MDT0001 complete [ 4050.036693] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 09:03:42 (1788440622) [ 4057.133803] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4057.626283] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4062.707604] LustreError: 134872:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4062.734509] LustreError: 134872:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4062.894846] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 4067.819224] LustreError: 134871:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4072.164456] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4072.531328] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4077.815314] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 4086.429801] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4092.941962] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4097.959119] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4097.970552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4097.979107] Lustre: Skipped 1 previous similar message [ 4109.279437] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4109.636098] Lustre: server umount lustre-MDT0000 complete [ 4113.376930] LustreError: 134868:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4113.396918] LustreError: 134868:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4113.419517] LustreError: 134851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440689 with bad export cookie 2668723956623328248 [ 4113.430248] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4113.436398] LustreError: 134851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4113.870884] Lustre: server umount lustre-MDT0001 complete [ 4122.679119] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 09:04:55 (1788440695) [ 4129.142087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4144.735753] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4149.856196] LustreError: 137288:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.157@tcp: failed processing log, type 4: rc = -110 [ 4175.583722] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4182.727233] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 4189.919587] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4191.815994] Lustre: server umount lustre-OST0000 complete [ 4201.088669] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 09:06:14 (1788440774) [ 4208.582977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4209.074678] LustreError: 138736:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4209.157408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4214.258633] LustreError: 138737:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4214.294487] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 4222.640956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4227.321746] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 4234.371832] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4240.234188] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4247.141194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4247.636763] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4247.643194] Lustre: Skipped 1 previous similar message [ 4254.398957] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 4263.467874] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4274.144854] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4274.164476] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4274.189600] Lustre: Skipped 1 previous similar message [ 4274.193891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4274.207878] Lustre: Skipped 4 previous similar messages [ 4277.302598] Lustre: server umount lustre-MDT0000 complete [ 4280.289420] LustreError: 138736:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4280.316615] LustreError: 138736:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4281.032147] LustreError: 138717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440856 with bad export cookie 2668723956623328920 [ 4281.040183] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4281.041146] LustreError: 138717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4281.522664] Lustre: server umount lustre-MDT0001 complete [ 4294.515853] Lustre: server umount lustre-OST0000 complete [ 4302.939671] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 09:07:56 (1788440876) [ 4309.387758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4309.900688] LustreError: 142391:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4314.468462] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 4322.029977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4322.289910] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4322.298294] Lustre: Skipped 1 previous similar message [ 4326.293916] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 4332.620806] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4338.412939] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4345.936112] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4353.552936] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 4354.686552] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4359.569614] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4364.073978] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4364.309181] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4369.059728] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4369.240697] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4373.473742] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4373.484547] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4373.517986] Lustre: Skipped 2 previous similar messages [ 4373.525368] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4373.536350] Lustre: Skipped 3 previous similar messages [ 4375.521240] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4377.531584] Lustre: server umount lustre-OST0000 complete [ 4383.721439] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4383.750640] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4383.776393] Lustre: Skipped 1 previous similar message [ 4387.129927] Lustre: server umount lustre-MDT0000 complete [ 4390.884033] LustreError: 142392:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4390.902578] LustreError: 142392:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4391.233257] LustreError: 145502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440967 with bad export cookie 2668723956623329991 [ 4391.243345] LustreError: 145502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4391.246933] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4391.764740] Lustre: server umount lustre-MDT0001 complete [ 4400.609368] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 09:09:33 (1788440973) [ 4407.775076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4423.327888] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4428.451683] LustreError: 146550:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.157@tcp: failed processing log, type 4: rc = -110 [ 4454.176462] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4454.183735] Lustre: Skipped 1 previous similar message [ 4461.260915] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 4468.804932] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4475.521701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4475.976610] LustreError: 146575:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4476.145122] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4480.539429] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 4487.817715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4492.539172] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 4499.700369] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4505.248750] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4509.737259] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4509.982395] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4514.396538] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4514.584344] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4518.880586] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4518.886773] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4518.903516] Lustre: Skipped 1 previous similar message [ 4518.907360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4518.914204] Lustre: Skipped 5 previous similar messages [ 4522.521669] Lustre: server umount lustre-MDT0000 complete [ 4525.840385] LustreError: 146556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441101 with bad export cookie 2668723956623331013 [ 4525.859720] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4526.297882] Lustre: server umount lustre-MDT0001 complete [ 4540.152283] Lustre: server umount lustre-OST0000 complete [ 4549.180911] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 09:12:01 (1788441121) [ 4558.020908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4573.791976] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4578.911535] LustreError: 150742:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.157@tcp: failed processing log, type 4: rc = -110 [ 4610.247239] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 4619.230953] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4625.714461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4626.226140] LustreError: 150766:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4626.245802] LustreError: 150766:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 4626.360888] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4631.428428] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 4640.440836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4645.563736] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 4652.915550] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4659.549648] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4667.536055] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4667.864535] Lustre: lustre-OST0001: new disk, initializing [ 4667.868263] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4667.875111] Lustre: 153617:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4674.147940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4674.172370] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4674.300967] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4675.793143] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 4684.793775] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4688.811175] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4688.960818] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4693.895249] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4694.130689] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4697.058975] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4697.067540] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4697.080932] Lustre: Skipped 2 previous similar messages [ 4697.087048] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4697.093992] Lustre: Skipped 3 previous similar messages [ 4710.367572] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4710.467479] Lustre: server umount lustre-OST0000 complete [ 4728.287131] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4728.535400] Lustre: server umount lustre-OST0001 complete [ 4732.897761] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4732.911340] LustreError: Skipped 3 previous similar messages [ 4738.516413] Lustre: server umount lustre-MDT0000 complete [ 4742.015929] LustreError: 150750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441317 with bad export cookie 2668723956623331965 [ 4742.017592] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4742.032205] LustreError: 150750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 4742.410351] Lustre: server umount lustre-MDT0001 complete [ 4763.966331] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 09:15:37 (1788441337) [ 4765.420730] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4767.282855] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4768.748404] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 09:15:42 (1788441342) [ 4775.365140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4775.669705] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4775.739448] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4775.945775] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4775.955491] Lustre: Skipped 6 previous similar messages [ 4780.652246] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 4788.644771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4788.720464] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4788.753696] Lustre: 157176:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4793.349355] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 4801.390911] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4806.590674] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4812.637176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4812.864233] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4819.572184] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4819.661812] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 4826.467529] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4831.112843] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4831.312734] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4836.044586] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4836.260213] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4839.904630] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4839.921599] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4839.944308] Lustre: Skipped 5 previous similar messages [ 4844.301960] Lustre: server umount lustre-OST0000 complete [ 4867.209891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4873.399906] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 4880.800567] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4880.962517] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4880.993194] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4885.552674] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4885.734609] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4890.118279] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4890.215334] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4890.232928] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4891.481741] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 1 sec [ 4896.743809] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8e86465f7000.ost_server_uuid 50 [ 4898.129590] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8e86465f7000.ost_server_uuid in FULL state after 0 sec [ 4898.310863] LustreError: 157928:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.203.57@tcp) returned error from blocking AST (req@00000000395e8451 x1875315757685760 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000a2ece847/0x2509364c8e1bdbd7 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.57@tcp remote: 0xe4c9643027b9383c expref: 5 pid: 157928 timeout: 4998 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4898.347667] LustreError: lustre-MDT0000: A client on nid 192.168.203.57@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4898.353424] LustreError: 157173:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.203.57@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000a2ece847/0x2509364c8e1bdbd7 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.57@tcp remote: 0xe4c9643027b9383c expref: 6 pid: 157928 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4908.936311] Lustre: server umount lustre-OST0000 complete [ 4919.107659] Lustre: server umount lustre-MDT0000 complete [ 4921.322619] LustreError: 157189:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4921.343722] LustreError: 157189:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 4922.945640] LustreError: 157169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441498 with bad export cookie 2668723956623333148 [ 4922.962610] LustreError: 157169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4922.964747] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4929.583206] Lustre: server umount lustre-MDT0001 complete [ 4941.544608] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 4944.476968] Key type lgssc unregistered [ 4944.816377] LNet: 163296:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4944.823632] LNetError: 163296:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4944.843966] LNet: Removed LNI 192.168.203.157@tcp [ 4945.696924] Key type .llcrypt unregistered [ 4945.705764] Key type ._llcrypt unregistered [ 4970.678288] Key type ._llcrypt registered [ 4970.684295] Key type .llcrypt registered [ 4970.806767] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 4971.847560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4971.881556] alg: No test for adler32 (adler32-zlib) [ 4973.061522] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 4973.299667] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 4974.959194] Key type lgssc registered [ 4976.199097] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4983.553116] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4983.567307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4985.107087] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4989.715903] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 4994.542949] LustreError: 164704:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4994.560834] LustreError: 164704:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4998.159573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4998.438152] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5002.699966] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 5009.682853] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5015.049515] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5020.997311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5021.267808] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5026.557872] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 5029.494800] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5034.484916] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5041.099641] Lustre: Failing over lustre-MDT0000 [ 5041.347560] Lustre: server umount lustre-MDT0000 complete [ 5042.415726] LustreError: 164701:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5044.707355] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5044.721639] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5044.741119] LustreError: 165434:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5045.229965] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5045.246396] Lustre: Skipped 1 previous similar message [ 5050.337046] LustreError: 165434:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5050.357142] LustreError: 165434:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 5053.625959] LustreError: 164686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.57@tcp arrived at 1788441629 with bad export cookie 12472070947141812966 [ 5053.643696] LustreError: 164686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5055.457687] LustreError: 164700:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5055.481690] LustreError: 164700:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5059.697114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5059.877804] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5060.115405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5060.174617] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5064.850285] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 5065.196544] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5065.216724] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5067.346516] Lustre: lustre-MDT0000: Denying connection for new client afaab1ad-d689-4ae0-b923-e00e621f378c (at 192.168.203.57@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 5080.032865] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5080.054880] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5080.094042] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5082.177244] Lustre: server umount lustre-OST0000 complete [ 5086.470075] LustreError: 168440:0:(ldlm_lib.c:3029:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5086.507558] Lustre: 167788:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5086.518699] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5086.552216] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5086.559690] LustreError: 167788:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000e1186476 x1875316885729024/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5086.570286] LustreError: 167788:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5086.577298] LustreError: 167788:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5086.584868] Lustre: lustre-MDT0000: Recovery over after 0:21, of 2 clients 0 recovered and 2 were evicted. [ 5087.128592] Lustre: server umount lustre-MDT0000 complete [ 5090.390276] LustreError: 164685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441666 with bad export cookie 12472070947141813064 [ 5090.395387] LustreError: 165434:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5090.397916] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5092.595952] Lustre: server umount lustre-MDT0001 complete [ 5102.199913] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 5104.891902] Key type lgssc unregistered [ 5105.182914] LNet: 169189:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5105.194870] LNetError: 169189:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5105.217726] LNet: Removed LNI 192.168.203.157@tcp [ 5106.011213] Key type .llcrypt unregistered [ 5106.015646] Key type ._llcrypt unregistered [ 5125.495885] Key type ._llcrypt registered [ 5125.499655] Key type .llcrypt registered [ 5125.605151] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 5126.154706] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5126.193598] alg: No test for adler32 (adler32-zlib) [ 5127.256636] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 5127.537497] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 5129.167168] Key type lgssc registered [ 5130.398208] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5138.306129] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5138.334930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5139.822610] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5145.318499] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 5149.156284] LustreError: 170576:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5149.186758] LustreError: 170576:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5154.273955] LustreError: 170577:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5155.541566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5156.034148] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5161.491766] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 5169.215718] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5174.941775] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5182.143644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5182.507400] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5184.578737] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5188.917830] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 5196.881564] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5204.960913] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5204.969234] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5204.988406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5205.990644] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5205.993091] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5206.010437] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5206.036176] Lustre: Skipped 1 previous similar message [ 5209.050678] Lustre: server umount lustre-OST0000 complete [ 5216.236973] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5216.250455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5216.252402] Lustre: Skipped 1 previous similar message [ 5216.265863] Lustre: Skipped 1 previous similar message [ 5219.281759] Lustre: server umount lustre-MDT0000 complete [ 5221.351388] LustreError: 171304:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5221.364621] LustreError: 171304:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5223.302416] LustreError: 170557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441799 with bad export cookie 445277850281075806 [ 5223.328514] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5223.332900] LustreError: 170557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5223.684983] Lustre: server umount lustre-MDT0001 complete [ 5234.828793] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 5238.089239] Key type lgssc unregistered [ 5238.498857] LNet: 174111:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5238.518317] LNetError: 174111:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5238.554178] LNet: Removed LNI 192.168.203.157@tcp [ 5239.394263] Key type .llcrypt unregistered [ 5239.397779] Key type ._llcrypt unregistered [ 5276.225608] Key type ._llcrypt registered [ 5276.228940] Key type .llcrypt registered [ 5276.373638] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 5277.593720] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5277.662973] alg: No test for adler32 (adler32-zlib) [ 5278.732378] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 5279.336483] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 5281.288783] Key type lgssc registered [ 5282.931144] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5291.552923] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5291.577872] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5293.237973] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5298.086452] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 5303.779641] LustreError: 175810:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5303.801072] LustreError: 175810:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5305.517503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5305.921480] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5310.712747] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 5318.586522] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5324.602349] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5331.779589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5332.076873] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5338.244641] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5339.525187] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 5346.559928] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5359.073895] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5359.087862] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5359.110179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5361.128325] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5361.133972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5365.346749] Lustre: server umount lustre-OST0000 complete [ 5371.361023] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5371.371925] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5371.380014] Lustre: Skipped 2 previous similar messages [ 5375.382700] Lustre: server umount lustre-MDT0000 complete [ 5376.489968] LustreError: 175806:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5376.505993] LustreError: 175806:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5379.111309] LustreError: 175790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441955 with bad export cookie 18257889603919837503 [ 5379.115678] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5379.122896] LustreError: 175790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5379.490278] Lustre: server umount lustre-MDT0001 complete [ 5390.304101] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 5393.398806] Key type lgssc unregistered [ 5393.701606] LNet: 179366:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5393.709907] LNetError: 179366:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5393.733293] LNet: Removed LNI 192.168.203.157@tcp [ 5394.612214] Key type .llcrypt unregistered [ 5394.617745] Key type ._llcrypt unregistered [ 5416.212569] Key type ._llcrypt registered [ 5416.215571] Key type .llcrypt registered [ 5416.318188] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 5417.096140] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5417.137063] alg: No test for adler32 (adler32-zlib) [ 5418.385258] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 5418.766332] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 5420.511162] Key type lgssc registered [ 5421.665504] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5433.072838] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5433.134466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5434.642792] Lustre: *** cfs_fail_loc=135, val=0*** [ 5434.729040] LustreError: 180884:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5434.740614] LustreError: 180884:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.157@tcp: cfg command failed: rc = -2 [ 5434.756527] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5434.756527] [ 5434.766774] LustreError: MGC192.168.203.157@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5434.783971] LustreError: 180846:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5434.800974] LustreError: 180846:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [ 5434.808067] LustreError: 180846:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 5434.813080] LustreError: 180846:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5434.867936] Lustre: server umount lustre-MDT0000 complete [ 5434.871295] LustreError: 180846:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5441.503022] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5441.930782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5446.089914] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 5451.234814] LustreError: 181255:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5451.251163] LustreError: 181255:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5454.877511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5455.292406] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5460.248809] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 5467.444147] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5473.982882] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5476.833791] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5476.838655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5476.855853] Lustre: Skipped 1 previous similar message [ 5476.872353] Lustre: Skipped 1 previous similar message [ 5480.998550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5481.007492] Lustre: Skipped 1 previous similar message [ 5482.354808] Lustre: server umount lustre-MDT0000 complete [ 5486.048738] LustreError: 181250:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5486.085372] LustreError: 181250:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5486.249460] LustreError: 181237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442062 with bad export cookie 5151782005416701063 [ 5486.253492] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5486.275714] LustreError: 181237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5486.640544] Lustre: server umount lustre-MDT0001 complete [ 5498.758274] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 5501.980867] Key type lgssc unregistered [ 5502.359836] LNet: 183578:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5502.379079] LNetError: 183578:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5502.413872] LNet: Removed LNI 192.168.203.157@tcp [ 5503.658973] Key type .llcrypt unregistered [ 5503.660954] Key type ._llcrypt unregistered [ 5525.724151] Key type ._llcrypt registered [ 5525.726963] Key type .llcrypt registered [ 5525.856898] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 5526.244337] Key type .llcrypt unregistered [ 5526.250276] Key type ._llcrypt unregistered [ 5541.841359] Key type ._llcrypt registered [ 5541.852941] Key type .llcrypt registered [ 5541.961366] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 5543.278359] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5543.352987] alg: No test for adler32 (adler32-zlib) [ 5544.455948] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 5544.746388] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 5546.471990] Key type lgssc registered [ 5547.648672] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5555.115145] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5555.148462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5570.783345] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5575.903456] LustreError: 185698:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.157@tcp: failed processing log, type 4: rc = -110 [ 5602.591254] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5608.892162] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 5616.867685] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5624.410486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5624.964280] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5625.022205] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5630.276249] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 5630.434175] LustreError: 185723:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5630.457530] LustreError: 185723:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5635.555299] LustreError: 185724:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5637.611342] LustreError: 186737:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5637.638233] LustreError: 186737:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5639.500422] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5639.774345] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5644.528565] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 5651.917211] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5658.311473] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5661.139181] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5675.491404] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5675.492020] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5675.506585] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5675.532880] Lustre: Skipped 1 previous similar message [ 5675.545072] Lustre: Skipped 1 previous similar message [ 5680.618153] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5680.626656] Lustre: Skipped 1 previous similar message [ 5685.758035] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5685.771024] Lustre: Skipped 1 previous similar message [ 5688.287453] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5688.573916] Lustre: server umount lustre-OST0000 complete [ 5695.969769] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5695.975816] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5695.992470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5696.001228] Lustre: Skipped 1 previous similar message [ 5699.336574] Lustre: server umount lustre-MDT0000 complete [ 5701.089871] LustreError: 186741:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5701.121076] LustreError: 186741:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5703.236091] LustreError: 185704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442279 with bad export cookie 12821989521224327672 [ 5703.239918] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5703.251907] LustreError: 185704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5704.010944] Lustre: server umount lustre-MDT0001 complete [ 5715.203555] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 5718.185366] Key type lgssc unregistered [ 5718.512786] LNet: 189713:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5718.525751] LNetError: 189713:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5718.553793] LNet: Removed LNI 192.168.203.157@tcp [ 5719.419565] Key type .llcrypt unregistered [ 5719.422579] Key type ._llcrypt unregistered [ 5742.096495] Key type ._llcrypt registered [ 5742.098505] Key type .llcrypt registered [ 5742.205308] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 5743.279040] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5743.329944] alg: No test for adler32 (adler32-zlib) [ 5744.570204] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 5745.008877] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 5746.759164] Key type lgssc registered [ 5748.070552] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5756.772184] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5756.783484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5758.199620] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5762.955398] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 5768.676322] LustreError: 191121:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5768.712178] LustreError: 191121:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5771.723735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5772.096946] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5775.843418] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 5782.684644] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5788.445335] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5794.894893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5795.337935] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5801.182367] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 5803.645462] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 5809.175748] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5814.789738] Lustre: Failing over lustre-MDT0000 [ 5815.066193] Lustre: server umount lustre-MDT0000 complete [ 5817.050665] LustreError: 192603:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5818.336477] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5818.348243] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5818.371747] LustreError: 191121:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5819.361583] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5819.379215] Lustre: Skipped 1 previous similar message [ 5822.124465] LustreError: 192603:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5822.145793] LustreError: 192603:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5827.253380] LustreError: 191116:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5827.276028] LustreError: 191116:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5833.616910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5833.761300] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5834.033393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5834.094819] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5837.359325] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5838.197834] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 5839.374096] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5839.403516] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 5839.456127] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 5842.779994] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 5850.859724] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 5856.738862] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5856.751778] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5856.772204] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5857.760824] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5857.773557] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5858.937909] Lustre: server umount lustre-OST0000 complete [ 5864.927937] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5864.936273] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5864.943075] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5864.962501] Lustre: Skipped 2 previous similar messages [ 5864.974141] Lustre: Skipped 1 previous similar message [ 5868.979577] Lustre: server umount lustre-MDT0000 complete [ 5870.058987] LustreError: 191117:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5870.070354] LustreError: 191117:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 5872.796344] LustreError: 191101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442448 with bad export cookie 5508924804267274639 [ 5872.804357] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5872.821256] LustreError: 191101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5873.254673] Lustre: server umount lustre-MDT0001 complete [ 5884.536633] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 5887.855343] Key type lgssc unregistered [ 5888.245905] LNet: 196006:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5888.259942] LNetError: 196006:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5888.292878] LNet: Removed LNI 192.168.203.157@tcp [ 5889.301260] Key type .llcrypt unregistered [ 5889.303995] Key type ._llcrypt unregistered [ 5914.079830] Key type ._llcrypt registered [ 5914.081690] Key type .llcrypt registered [ 5914.201966] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 5914.903434] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5914.975091] alg: No test for adler32 (adler32-zlib) [ 5916.318367] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 5916.561639] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 5918.223214] Key type lgssc registered [ 5919.385929] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5926.768710] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5926.801837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5928.369922] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5932.774987] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 5938.659863] LustreError: 197414:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5938.674896] LustreError: 197414:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5941.358978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5941.791474] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5946.493950] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 5954.878393] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5961.488674] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5968.481385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5968.955143] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5974.072222] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 5975.535429] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 5983.371914] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5986.585488] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 5991.083883] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6006.624879] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6014.944163] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6014.951702] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6014.974066] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6017.000547] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6017.017369] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6017.040489] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6018.878161] Lustre: server umount lustre-OST0000 complete [ 6023.650935] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6023.656051] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6023.672445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6023.677872] Lustre: Skipped 1 previous similar message [ 6029.246323] Lustre: server umount lustre-MDT0000 complete [ 6030.304772] LustreError: 200016:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6030.330051] LustreError: 200016:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6033.440355] LustreError: 200216:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442609 with bad export cookie 15313710440338947986 [ 6033.443161] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6033.447259] LustreError: 200216:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6033.716683] Lustre: server umount lustre-MDT0001 complete [ 6045.360804] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 6049.262262] Key type lgssc unregistered [ 6049.556220] LNet: 201270:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6049.566925] LNetError: 201270:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6049.596628] LNet: Removed LNI 192.168.203.157@tcp [ 6051.102078] Key type .llcrypt unregistered [ 6051.104845] Key type ._llcrypt unregistered [ 6068.599368] Key type ._llcrypt registered [ 6068.602548] Key type .llcrypt registered [ 6068.758934] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 6069.696364] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6069.753159] alg: No test for adler32 (adler32-zlib) [ 6070.782945] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 6070.974627] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 6072.615249] Key type lgssc registered [ 6073.586920] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6080.556187] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6080.571397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6082.037218] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6086.552698] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 6091.235712] LustreError: 202532:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6091.265960] LustreError: 202532:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6094.365508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6094.823592] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6099.431595] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 6109.664585] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6116.826733] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6123.961092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6124.363394] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6126.469428] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6130.125505] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 6138.325112] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6143.505049] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6152.996671] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6161.022529] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6168.518285] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6175.715508] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6175.717298] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6175.727877] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6175.736371] Lustre: Skipped 1 previous similar message [ 6180.839153] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6180.846576] Lustre: Skipped 1 previous similar message [ 6185.961948] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6185.966348] Lustre: Skipped 1 previous similar message [ 6189.023829] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6189.173442] Lustre: server umount lustre-OST0000 complete [ 6196.195343] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6196.198993] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6196.213186] Lustre: Skipped 2 previous similar messages [ 6198.937410] Lustre: server umount lustre-MDT0000 complete [ 6201.312568] LustreError: 202532:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6201.337433] LustreError: 202532:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6202.384481] LustreError: 202512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442778 with bad export cookie 3991542100799946255 [ 6202.394673] LustreError: 202512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6202.395315] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6202.738324] Lustre: server umount lustre-MDT0001 complete [ 6213.366956] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 6216.099421] Key type lgssc unregistered [ 6216.394673] LNet: 207363:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6216.400118] LNetError: 207363:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6216.426791] LNet: Removed LNI 192.168.203.157@tcp [ 6217.327097] Key type .llcrypt unregistered [ 6217.329202] Key type ._llcrypt unregistered [ 6240.180348] Key type ._llcrypt registered [ 6240.183112] Key type .llcrypt registered [ 6240.302886] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 6241.614801] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6241.806257] alg: No test for adler32 (adler32-zlib) [ 6242.960465] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 6243.203067] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 6244.895151] Key type lgssc registered [ 6245.869188] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6253.882346] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6253.926427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6255.298201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6260.614521] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 6265.828830] LustreError: 208771:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6265.842265] LustreError: 208771:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6268.956607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6269.492946] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6274.350247] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 6281.215191] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6286.670953] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6294.300703] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6294.803246] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6301.016794] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6302.727728] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 6311.604399] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6326.755824] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6326.757026] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6326.762667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6326.768246] Lustre: Skipped 1 previous similar message [ 6329.981321] Lustre: server umount lustre-OST0000 complete [ 6335.968585] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6335.981606] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6336.017639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6336.023247] Lustre: Skipped 1 previous similar message [ 6336.999325] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6337.022165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6337.024196] Lustre: Skipped 1 previous similar message [ 6339.493657] Lustre: server umount lustre-MDT0000 complete [ 6342.122343] LustreError: 208766:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6342.150044] LustreError: 208766:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6344.797127] LustreError: 208752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442920 with bad export cookie 9317657810109881913 [ 6344.804242] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6344.825019] LustreError: 208752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6345.431440] Lustre: server umount lustre-MDT0001 complete [ 6358.994023] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 6361.174460] Key type lgssc unregistered [ 6361.368171] LNet: 212452:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6361.384027] LNetError: 212452:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6361.400822] LNet: Removed LNI 192.168.203.157@tcp [ 6362.589540] Key type .llcrypt unregistered [ 6362.593286] Key type ._llcrypt unregistered [ 6381.213449] Key type ._llcrypt registered [ 6381.218933] Key type .llcrypt registered [ 6381.321515] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 6382.253188] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6382.407605] alg: No test for adler32 (adler32-zlib) [ 6383.548581] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 6383.778192] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 6385.463218] Key type lgssc registered [ 6386.895906] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6395.805056] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6395.824652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6397.563202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6403.161042] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 6407.149052] LustreError: 213715:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6407.172916] LustreError: 213715:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6412.265192] LustreError: 213716:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6412.867703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6413.298411] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6417.767400] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 6425.777691] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6431.056408] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6438.755187] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6438.884890] LustreError: 215536:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6439.110361] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6442.165991] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6445.223351] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 6452.808027] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6468.073894] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6468.078626] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6468.083722] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6468.101187] Lustre: Skipped 1 previous similar message [ 6468.124672] LustreError: Skipped 1 previous similar message [ 6473.188925] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6473.201645] Lustre: Skipped 2 previous similar messages [ 6473.682255] Lustre: server umount lustre-OST0000 complete [ 6478.330375] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6478.364403] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6478.366776] Lustre: Skipped 1 previous similar message [ 6478.398422] Lustre: Skipped 1 previous similar message [ 6483.426413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6483.435567] Lustre: Skipped 1 previous similar message [ 6484.349460] Lustre: server umount lustre-MDT0000 complete [ 6488.372399] LustreError: 213697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443064 with bad export cookie 10687972199138187732 [ 6488.374038] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6488.383592] LustreError: 213697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6488.959540] Lustre: server umount lustre-MDT0001 complete [ 6499.266759] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 6502.019556] Key type lgssc unregistered [ 6502.376612] LNet: 217399:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6502.387573] LNetError: 217399:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6502.409756] LNet: Removed LNI 192.168.203.157@tcp [ 6503.273262] Key type .llcrypt unregistered [ 6503.276678] Key type ._llcrypt unregistered [ 6524.142146] Key type ._llcrypt registered [ 6524.148251] Key type .llcrypt registered [ 6524.242556] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 6524.910737] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6525.007768] alg: No test for adler32 (adler32-zlib) [ 6526.102485] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 6526.505718] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 6528.191199] Key type lgssc registered [ 6529.449199] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6537.488828] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6537.531427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6539.072595] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6543.815818] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 6548.451760] LustreError: 218661:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6548.486734] LustreError: 218661:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6551.668834] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6552.111932] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6556.656766] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 6564.644199] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6570.750584] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6577.402573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6577.889364] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6583.563528] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 6584.062575] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6592.219768] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6602.269857] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6602.351745] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6602.591562] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6609.193850] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 6616.501084] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6618.183117] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6629.925401] Lustre: Permanently deactivating lustre-OST0001 [ 6629.949331] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6639.705322] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6648.678745] Lustre: Permanently reactivating lustre-OST0001 [ 6648.690604] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6648.698774] Lustre: Skipped 2 previous similar messages [ 6657.085443] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6657.102169] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6657.108897] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6657.125313] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6657.169396] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6659.559977] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6659.577048] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6659.579417] Lustre: Skipped 2 previous similar messages [ 6659.596053] Lustre: Skipped 1 previous similar message [ 6659.759541] Lustre: server umount lustre-OST0001 complete [ 6663.136698] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6663.145183] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6663.170312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6664.683533] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6664.684444] LustreError: 220485:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6664.693932] Lustre: Skipped 1 previous similar message [ 6664.728752] LustreError: 220485:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6669.367436] Lustre: server umount lustre-OST0000 complete [ 6674.917352] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6674.918273] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6674.929630] Lustre: Skipped 1 previous similar message [ 6674.931694] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6674.951164] Lustre: Skipped 1 previous similar message [ 6679.651227] Lustre: server umount lustre-MDT0000 complete [ 6680.033738] LustreError: 218660:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6680.048234] LustreError: 218660:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6683.157527] LustreError: 223018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443259 with bad export cookie 10057411703756971896 [ 6683.159231] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6683.168565] LustreError: 223018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6683.512560] Lustre: server umount lustre-MDT0001 complete [ 6694.113312] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 6696.844528] Key type lgssc unregistered [ 6697.160816] LNet: 223769:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6697.176806] LNetError: 223769:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6697.197685] LNet: Removed LNI 192.168.203.157@tcp [ 6698.009528] Key type .llcrypt unregistered [ 6698.012799] Key type ._llcrypt unregistered [ 6734.241329] Key type ._llcrypt registered [ 6734.243191] Key type .llcrypt registered [ 6734.332911] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 6735.411997] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6735.477148] alg: No test for adler32 (adler32-zlib) [ 6736.594288] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 6736.935813] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 6738.617760] Key type lgssc registered [ 6739.723137] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6747.056214] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6747.079458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6748.326484] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6748.352645] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6748.523849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6752.699840] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 6760.571111] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6760.681245] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6760.718573] Lustre: 225966:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6761.062539] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6766.090758] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 6772.884986] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6777.951517] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6785.317978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6785.534509] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6785.734358] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6791.839684] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 6794.313584] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 6799.114155] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6802.822972] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6808.268322] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6817.769764] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6824.558044] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6838.664627] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6838.670715] Lustre: Skipped 1 previous similar message [ 6856.889804] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6856.895537] Lustre: Skipped 1 previous similar message [ 6896.107290] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6896.117301] Lustre: Skipped 4 previous similar messages [ 6970.922658] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 6970.932251] Lustre: Skipped 8 previous similar messages [ 6978.532164] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6978.545666] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6978.559113] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6978.566918] Lustre: Skipped 1 previous similar message [ 6983.660398] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6983.666522] Lustre: Skipped 1 previous similar message [ 6984.647327] Lustre: server umount lustre-OST0000 complete [ 6988.770569] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6988.793943] Lustre: Skipped 1 previous similar message [ 6988.802441] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6993.904704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6993.909854] Lustre: Skipped 2 previous similar messages [ 6999.010930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6999.028939] Lustre: Skipped 1 previous similar message [ 7003.103953] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7003.243723] Lustre: server umount lustre-MDT0000 complete [ 7008.064196] LustreError: 225961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443583 with bad export cookie 2868824654723942196 [ 7008.070385] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7008.100875] LustreError: 225961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7008.801378] Lustre: server umount lustre-MDT0001 complete [ 7022.439575] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 7025.588031] Key type lgssc unregistered [ 7025.888361] LNet: 230785:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7025.897878] LNetError: 230785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7025.916898] LNet: Removed LNI 192.168.203.157@tcp [ 7026.631910] Key type .llcrypt unregistered [ 7026.633756] Key type ._llcrypt unregistered [ 7046.850950] Key type ._llcrypt registered [ 7046.853700] Key type .llcrypt registered [ 7047.041526] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 7048.488384] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7048.645219] alg: No test for adler32 (adler32-zlib) [ 7049.808281] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 7050.108081] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 7051.815302] Key type lgssc registered [ 7053.274102] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7061.941564] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7061.967867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7063.539652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7068.495936] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 7073.766038] LustreError: 232174:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7073.790637] LustreError: 232174:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7076.485874] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7076.753644] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7082.226865] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 7090.988974] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7096.437618] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7103.592530] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7103.997595] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7108.162909] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7111.410383] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 7121.668747] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7138.784252] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7138.788111] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7138.797980] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7140.478085] Lustre: server umount lustre-OST0000 complete [ 7146.979839] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7146.994071] Lustre: Skipped 1 previous similar message [ 7146.997850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7147.029541] Lustre: Skipped 1 previous similar message [ 7151.643486] Lustre: server umount lustre-MDT0000 complete [ 7152.100712] LustreError: 232902:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7152.155324] LustreError: 232902:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7156.020478] LustreError: 232153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443731 with bad export cookie 6313394751171119157 [ 7156.032516] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7156.047883] LustreError: 232153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7165.434646] Lustre: server umount lustre-MDT0001 complete [ 7176.572546] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 7179.378880] Key type lgssc unregistered [ 7179.703812] LNet: 235866:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7179.712898] LNetError: 235866:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7179.740811] LNet: Removed LNI 192.168.203.157@tcp [ 7180.605163] Key type .llcrypt unregistered [ 7180.610904] Key type ._llcrypt unregistered [ 7208.189234] Key type ._llcrypt registered [ 7208.194876] Key type .llcrypt registered [ 7208.342402] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 7209.593577] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7209.700072] alg: No test for adler32 (adler32-zlib) [ 7210.651949] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 7210.934552] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 7212.575227] Key type lgssc registered [ 7213.651146] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7486.722689] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 7489.684596] Key type lgssc unregistered [ 7490.082716] LNet: 237897:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7490.104697] LNetError: 237897:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7490.156638] LNet: Removed LNI 192.168.203.157@tcp [ 7491.127706] Key type .llcrypt unregistered [ 7491.132983] Key type ._llcrypt unregistered [ 7519.201498] Key type ._llcrypt registered [ 7519.203594] Key type .llcrypt registered [ 7519.386066] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_hostid [ 7540.006973] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 7541.637052] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7541.726592] alg: No test for adler32 (adler32-zlib) [ 7542.940187] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 7543.298282] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 7544.967222] Key type lgssc registered [ 7546.490402] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7583.255499] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7308 sec ========== 10:02:36 (1788444156) [ 7584.620614] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 10:02:37 (1788444157) === [ 7588.838675] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 10:02:41 (1788444161) === [ 7632.689046] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 7635.731151] Key type lgssc unregistered [ 7636.093141] LNet: 243131:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7636.102837] LNetError: 243131:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7636.120736] LNet: Removed LNI 192.168.203.157@tcp [ 7637.033249] Key type .llcrypt unregistered [ 7637.037154] Key type ._llcrypt unregistered