[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 486290875 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003195] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.008748] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010011] pid_max: default: 32768 minimum: 301 [ 0.011143] LSM: Security Framework initializing [ 0.012084] Yama: becoming mindful. [ 0.013067] SELinux: Initializing. [ 0.014104] *** VALIDATE selinux *** [ 0.023687] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028377] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030020] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032069] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033158] *** VALIDATE tmpfs *** [ 0.034566] *** VALIDATE proc *** [ 0.035353] *** VALIDATE cgroup *** [ 0.036021] *** VALIDATE cgroup2 *** [ 0.038289] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.040131] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.041015] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.042040] Spectre V2 : User space: Vulnerable [ 0.043013] Speculative Store Bypass: Vulnerable [ 0.047234] debug: unmapping init [mem 0xffffffffa8259000-0xffffffffa8260fff] [ 0.049225] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.050927] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.051031] ... version: 2 [ 0.052021] ... bit width: 48 [ 0.053022] ... generic registers: 4 [ 0.054021] ... value mask: 0000ffffffffffff [ 0.055025] ... max period: 00007fffffffffff [ 0.056023] ... fixed-purpose events: 3 [ 0.057016] ... event mask: 000000070000000f [ 0.058444] rcu: Hierarchical SRCU implementation. [ 0.060769] smp: Bringing up secondary CPUs ... [ 0.061800] x86: Booting SMP configuration: [ 0.062042] .... node #0, CPUs: #1 #2 #3 [ 0.066021] smp: Brought up 1 node, 4 CPUs [ 0.068022] smpboot: Max logical packages: 1 [ 0.069027] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.233104] node 0 deferred pages initialised in 162ms [ 0.236148] devtmpfs: initialized [ 0.237292] x86/mm: Memory block size: 128MB [ 0.239940] gcov: version magic: 0x41383552 [ 0.241380] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.242113] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.243335] pinctrl core: initialized pinctrl subsystem [ 0.244288] [ 0.244952] ************************************************************* [ 0.245023] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.246023] ** ** [ 0.247020] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.248023] ** ** [ 0.249019] ** This means that this kernel is built to expose internal ** [ 0.250021] ** IOMMU data structures, which may compromise security on ** [ 0.251024] ** your system. ** [ 0.252021] ** ** [ 0.253022] ** If you see this message and you are not debugging the ** [ 0.254023] ** kernel, report this immediately to your vendor! ** [ 0.255017] ** ** [ 0.256024] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.257024] ************************************************************* [ 0.258753] NET: Registered protocol family 16 [ 0.259543] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.260103] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.261100] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.262519] cpuidle: using governor menu [ 0.265246] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.268559] PCI: Using configuration type 1 for base access [ 0.271131] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.280228] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.281032] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.283038] cryptd: max_cpu_qlen set to 1000 [ 0.285290] ACPI: Added _OSI(Module Device) [ 0.287015] ACPI: Added _OSI(Processor Device) [ 0.290017] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.292029] ACPI: Added _OSI(Processor Aggregator Device) [ 0.297248] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.305509] ACPI: Interpreter enabled [ 0.308053] ACPI: PM: (supports S0 S3 S4 S5) [ 0.309016] ACPI: Using IOAPIC for interrupt routing [ 0.311112] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.315397] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.326324] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.328063] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.331022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.335080] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.341443] acpiphp: Slot [2] registered [ 0.343168] acpiphp: Slot [5] registered [ 0.344130] acpiphp: Slot [6] registered [ 0.345124] acpiphp: Slot [7] registered [ 0.347164] acpiphp: Slot [8] registered [ 0.349209] acpiphp: Slot [9] registered [ 0.351180] acpiphp: Slot [10] registered [ 0.353171] acpiphp: Slot [3] registered [ 0.354092] acpiphp: Slot [4] registered [ 0.355189] acpiphp: Slot [11] registered [ 0.357114] acpiphp: Slot [12] registered [ 0.359126] acpiphp: Slot [13] registered [ 0.360095] acpiphp: Slot [14] registered [ 0.362110] acpiphp: Slot [15] registered [ 0.363129] acpiphp: Slot [16] registered [ 0.365169] acpiphp: Slot [17] registered [ 0.366110] acpiphp: Slot [18] registered [ 0.368098] acpiphp: Slot [19] registered [ 0.369087] acpiphp: Slot [20] registered [ 0.370107] acpiphp: Slot [21] registered [ 0.372115] acpiphp: Slot [22] registered [ 0.374120] acpiphp: Slot [23] registered [ 0.375075] acpiphp: Slot [24] registered [ 0.376086] acpiphp: Slot [25] registered [ 0.378079] acpiphp: Slot [26] registered [ 0.379094] acpiphp: Slot [27] registered [ 0.380062] acpiphp: Slot [28] registered [ 0.382101] acpiphp: Slot [29] registered [ 0.383097] acpiphp: Slot [30] registered [ 0.385110] acpiphp: Slot [31] registered [ 0.386060] PCI host bridge to bus 0000:00 [ 0.389024] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.391021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.393026] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.395025] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.398027] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.401030] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.403189] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.406071] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.409408] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.419805] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.425054] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.427017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.429021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.432020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.435455] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.439786] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.442059] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.446018] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.452022] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.467023] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.474011] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.480000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.487021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.497017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.520017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.529912] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.535017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.542029] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.558018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.565870] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.573018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.582022] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.598018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.610287] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.616015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.622014] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.638014] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.648683] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.656019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.663017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.679020] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.688725] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.696016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.704017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.722018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.736576] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.741974] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.745370] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.748389] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.750288] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.756044] iommu: Default domain type: Passthrough [ 0.758415] SCSI subsystem initialized [ 0.760148] ACPI: bus type USB registered [ 0.761201] usbcore: registered new interface driver usbfs [ 0.764089] usbcore: registered new interface driver hub [ 0.766086] usbcore: registered new device driver usb [ 0.768156] pps_core: LinuxPPS API ver. 1 registered [ 0.770015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.773158] PTP clock support registered [ 0.775240] EDAC MC: Ver: 3.0.0 [ 0.777106] PCI: Using ACPI for IRQ routing [ 0.778585] NetLabel: Initializing [ 0.780018] NetLabel: domain hash size = 128 [ 0.782013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.784081] NetLabel: unlabeled traffic allowed by default [ 0.787196] vgaarb: loaded [ 0.789400] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.791018] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.797765] clocksource: Switched to clocksource kvm-clock [ 0.905929] VFS: Disk quotas dquot_6.6.0 [ 0.907835] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.910699] *** VALIDATE ramfs *** [ 0.912340] *** VALIDATE hugetlbfs *** [ 0.914615] pnp: PnP ACPI init [ 0.917231] pnp: PnP ACPI: found 6 devices [ 0.938514] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.942533] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.944865] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.947581] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.950350] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.953139] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.956791] NET: Registered protocol family 2 [ 0.959260] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.963961] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.967386] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.974119] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.978831] TCP: Hash tables configured (established 65536 bind 65536) [ 0.982905] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.987126] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.990869] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.994818] NET: Registered protocol family 1 [ 0.997754] RPC: Registered named UNIX socket transport module. [ 0.999908] RPC: Registered udp transport module. [ 1.001497] RPC: Registered tcp transport module. [ 1.002963] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.004812] NET: Registered protocol family 44 [ 1.006262] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.008483] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.010698] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.013168] PCI: CLS 0 bytes, default 64 [ 1.015143] Unpacking initramfs... [ 2.450258] debug: unmapping init [mem 0xffff8b5a3cc54000-0xffff8b5a3ffbffff] [ 2.454438] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.456277] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.459095] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.963540] Initialise system trusted keyrings [ 2.966095] Key type blacklist registered [ 2.967963] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.977921] zbud: loaded [ 2.981138] *** VALIDATE nfs *** [ 2.982904] *** VALIDATE nfs4 *** [ 2.985057] pstore: using deflate compression [ 2.990204] Platform Keyring initialized [ 3.107392] NET: Registered protocol family 38 [ 3.109114] Key type asymmetric registered [ 3.110320] Asymmetric key parser 'x509' registered [ 3.112256] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.116336] io scheduler mq-deadline registered [ 3.117917] io scheduler kyber registered [ 3.119276] io scheduler bfq registered [ 3.121197] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.124126] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.126901] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.130110] ACPI: Power Button [PWRF] [ 3.135900] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.142783] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.156223] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.164479] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.183029] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.211445] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.240774] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.246630] Non-volatile memory driver v1.3 [ 3.248562] Linux agpgart interface v0.103 [ 3.280842] virtio_blk virtio1: [vda] 149760 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.283919] vda: detected capacity change from 0 to 76677120 [ 3.299343] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.302850] vdb: detected capacity change from 0 to 1073741824 [ 3.318336] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.321628] vdc: detected capacity change from 0 to 2621440000 [ 3.335829] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.340030] vdd: detected capacity change from 0 to 2621440000 [ 3.356843] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.359772] vde: detected capacity change from 0 to 4294967296 [ 3.375299] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.378596] vdf: detected capacity change from 0 to 4294967296 [ 3.385866] libphy: Fixed MDIO Bus: probed [ 3.392475] usbcore: registered new interface driver usbserial_generic [ 3.395456] usbserial: USB Serial support registered for generic [ 3.398227] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.403415] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.405707] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.408076] mousedev: PS/2 mouse device common for all mice [ 3.410819] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.412701] rtc_cmos 00:05: RTC can wake from S4 [ 3.418318] rtc_cmos 00:05: registered as rtc0 [ 3.418985] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.420861] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.428451] intel_pstate: CPU model not supported [ 3.431423] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.436503] hid: raw HID events driver (C) Jiri Kosina [ 3.439461] usbcore: registered new interface driver usbhid [ 3.441498] usbhid: USB HID core driver [ 3.443372] drop_monitor: Initializing network drop monitor service [ 3.446353] Initializing XFRM netlink socket [ 3.448804] NET: Registered protocol family 10 [ 3.452136] Segment Routing with IPv6 [ 3.453909] NET: Registered protocol family 17 [ 3.456256] mpls_gso: MPLS GSO support [ 3.462323] RAS: Correctable Errors collector initialized. [ 3.464473] AVX version of gcm_enc/dec engaged. [ 3.466210] AES CTR mode by8 optimization enabled [ 3.543509] sched_clock: Marking stable (3543485874, 0)->(4468650863, -925164989) [ 3.548541] registered taskstats version 1 [ 3.550888] Loading compiled-in X.509 certificates [ 3.553053] zswap: loaded using pool lzo/zbud [ 3.579159] Key type big_key registered [ 3.592947] Key type encrypted registered [ 3.594991] ima: No TPM chip found, activating TPM-bypass! [ 3.596956] ima: Allocated hash algorithm: sha1 [ 3.598798] ima: No architecture policies found [ 3.600596] evm: Initialising EVM extended attributes: [ 3.603486] evm: security.selinux [ 3.604957] evm: security.ima [ 3.606197] evm: security.capability [ 3.607736] evm: HMAC attrs: 0x1 [ 3.610168] rtc_cmos 00:05: setting system clock to 2026-09-03 12:40:32 UTC (1788439232) [ 3.629237] debug: unmapping init [mem 0xffffffffa9203000-0xffffffffa93fffff] [ 3.632327] debug: unmapping init [mem 0xffffffffa7f82000-0xffffffffa8258fff] [ 3.645083] Write protecting the kernel read-only data: 28672k [ 3.648859] debug: unmapping init [mem 0xffffffffa6603000-0xffffffffa67fffff] [ 3.652015] debug: unmapping init [mem 0xffffffffa6f14000-0xffffffffa6ffffff] [ 3.710689] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.719859] systemd[1]: Detected virtualization kvm. [ 3.722577] systemd[1]: Detected architecture x86-64. [ 3.724639] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.755495] systemd[1]: No hostname configured. [ 3.757664] systemd[1]: Set hostname to . [ 3.760153] random: systemd: uninitialized urandom read (16 bytes read) [ 3.763097] systemd[1]: Initializing machine ID from random generator. [ 3.815180] random: ln: uninitialized urandom read (6 bytes read) [ 3.916229] random: systemd: uninitialized urandom read (16 bytes read) [ 3.918973] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.924815] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.929670] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Kernel Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Reached target Swap. [ OK ] Reached target Slices. [ OK ] Reached target Initrd Root Device. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.559000] device-mapper: uevent: version 1.0.3 [ 4.561383] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.324304] virtio_net virtio0 ens2: renamed from eth0 [ 5.355419] random: fast init done [ 5.381840] scsi host0: ata_piix [ 5.396824] scsi host1: ata_piix [ 5.398595] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.400671] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.893605] dracut-initqueue[590]: RTNETLINK answers: File exists [ 10.112365] random: crng init done [ 10.113834] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.668448] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.847516] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.135895] SELinux: Disabled at runtime. [ 12.200088] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.211602] systemd[1]: Detected virtualization kvm. [ 12.215987] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.773696] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.777687] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.785943] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.790361] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.794528] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.802381] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.811380] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-getty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target rpc_pipefs.target. Mounting Huge Pages File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ 12.889289] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Control Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting POSIX Message Queue File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on Process Core Dump Socket. Starting udev Coldplug all Devices... Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.325322] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.687750] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.708983] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.901343] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.911867] EDAC sbridge: Ver: 1.1.2 [ 15.866786] Key type dns_resolver registered [ 16.173958] NFS: Registering the id_resolver key type [ 16.175502] Key type id_resolver registered [ 16.176620] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg322-server login: [ 42.439887] libcfs: loading out-of-tree module taints kernel. [ 42.457241] Key type ._llcrypt registered [ 42.458875] Key type .llcrypt registered [ 42.513523] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 63.505190] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 65.062842] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 65.099563] alg: No test for adler32 (adler32-zlib) [ 66.660989] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 67.648276] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 69.479922] Key type lgssc registered [ 71.698331] Lustre: Echo OBD driver; http://www.lustre.org/ [ 73.170047] hrtimer: interrupt took 6039425 ns [ 91.969980] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 134.757975] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 147.123228] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 147.158857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 148.381222] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 148.415975] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 148.529111] Lustre: lustre-MDT0000: new disk, initializing [ 148.634813] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 148.674214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 153.339539] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 167.275828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 167.404679] Lustre: 6519:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 167.437209] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 167.443174] Lustre: Skipped 1 previous similar message [ 167.623649] Lustre: lustre-MDT0001: new disk, initializing [ 167.707251] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 167.741962] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 167.755513] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 172.280834] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 176.900598] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 185.955887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 186.200371] Lustre: lustre-OST0000: new disk, initializing [ 186.203950] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 186.213979] Lustre: 8456:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 186.309742] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 192.813805] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 194.145944] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 194.169608] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 194.309891] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 207.066206] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 207.195333] Lustre: lustre-OST0001: new disk, initializing [ 207.198581] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 207.202954] Lustre: 9528:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 207.257906] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 213.521233] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 213.548533] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 213.559415] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 213.631307] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 226.440886] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 235.437862] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 242.101715] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing check_logdir /tmp/testlogs/ [ 247.032549] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing yml_node [ 251.644470] Lustre: DEBUG MARKER: Client: 2.17.58.2 [ 254.707396] Lustre: DEBUG MARKER: MDS: 2.17.58.2 [ 258.095874] Lustre: DEBUG MARKER: OSS: 2.17.58.2 [ 260.541523] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 3 08:44:46 EDT 2026 [ 280.636891] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 282.296370] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 300.512592] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 300.513774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 300.557348] Lustre: Skipped 2 previous similar messages [ 300.590414] Lustre: Skipped 3 previous similar messages [ 304.790319] Lustre: server umount lustre-MDT0000 complete [ 309.149458] LustreError: 6509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439538 with bad export cookie 18114427855193280322 [ 309.158935] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 309.162231] LustreError: 6509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 310.752205] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 310.757969] Lustre: Skipped 2 previous similar messages [ 310.760197] LustreError: 6526:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 310.760759] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 310.769663] LustreError: 6526:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 310.786326] Lustre: Skipped 1 previous similar message [ 314.784322] LustreError: 9536:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 314.785340] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 314.813086] LustreError: 9536:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 314.822250] Lustre: Skipped 1 previous similar message [ 315.487302] Lustre: server umount lustre-MDT0001 complete [ 318.840027] Lustre: server umount lustre-OST0000 complete [ 322.245906] Lustre: server umount lustre-OST0001 complete [ 328.235773] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 337.365217] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 373.651165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 373.928829] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 374.001332] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 374.138226] Lustre: lustre-MDT0000: new disk, initializing [ 374.328497] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 374.372250] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 379.603503] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 390.797880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 390.950683] Lustre: 15081:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 391.061731] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 391.070967] Lustre: Skipped 1 previous similar message [ 391.186796] Lustre: lustre-MDT0001: new disk, initializing [ 391.275077] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 391.326253] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 391.340328] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 395.971660] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 406.323311] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 412.227817] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 418.926430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 419.169394] Lustre: lustre-OST0000: new disk, initializing [ 419.173204] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 419.179824] Lustre: 17051:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 419.278846] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 421.236152] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 421.250735] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 421.324833] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 425.461690] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 435.387325] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 440.054469] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 440.232246] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 444.864722] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 445.080816] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 446.949675] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 446.966409] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 446.968387] LustreError: Skipped 1 previous similar message [ 446.997930] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 447.015949] Lustre: Skipped 1 previous similar message [ 452.068631] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 452.079576] Lustre: Skipped 1 previous similar message [ 461.279166] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 461.501648] Lustre: server umount lustre-OST0000 complete [ 467.428432] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 467.460223] Lustre: Skipped 1 previous similar message [ 467.472837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 467.484903] Lustre: Skipped 3 previous similar messages [ 471.417425] Lustre: server umount lustre-MDT0000 complete [ 472.544857] LustreError: 15092:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 472.561807] LustreError: 15092:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 475.691370] LustreError: 15073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439704 with bad export cookie 18114427855193281855 [ 475.697522] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 475.715866] LustreError: 15073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 476.254849] Lustre: server umount lustre-MDT0001 complete [ 479.739606] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 08:48:26 (1788439706) [ 486.400922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 486.790231] LustreError: 19317:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 486.854208] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 491.092321] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 492.003208] LustreError: 19318:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 499.464337] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 503.994858] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 512.241541] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 517.624601] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 526.134995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 533.164285] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 540.750623] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 551.912078] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 551.926292] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 551.946238] Lustre: Skipped 1 previous similar message [ 551.955391] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 565.215140] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 565.443170] Lustre: server umount lustre-OST0000 complete [ 570.855329] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 570.873421] Lustre: Skipped 1 previous similar message [ 576.401740] Lustre: server umount lustre-MDT0000 complete [ 580.209447] LustreError: 22301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439809 with bad export cookie 18114427855193282730 [ 580.217987] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 580.218369] LustreError: 22301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 580.708801] Lustre: server umount lustre-MDT0001 complete [ 593.877772] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 597.023021] Key type lgssc unregistered [ 597.392334] LNet: 22850:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 597.403344] LNetError: 22850:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 597.430983] LNet: Removed LNI 192.168.203.122@tcp [ 598.684798] Key type .llcrypt unregistered [ 598.690694] Key type ._llcrypt unregistered [ 624.741342] Key type ._llcrypt registered [ 624.743838] Key type .llcrypt registered [ 624.817983] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 625.869657] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 625.907162] alg: No test for adler32 (adler32-zlib) [ 627.520639] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 627.964810] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 629.759196] Key type lgssc registered [ 631.103593] Lustre: Echo OBD driver; http://www.lustre.org/ [ 639.210231] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 639.226376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 640.814475] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 646.086362] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 650.217664] LustreError: 24263:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 650.251472] LustreError: 24263:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 655.333665] LustreError: 24264:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 655.677174] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 656.070883] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 660.546274] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 668.062973] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 673.593949] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 680.163488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 680.533170] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 685.683373] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 686.305101] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 693.629974] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 710.849154] Lustre: server umount lustre-OST0000 complete [ 716.770365] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 716.783037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 716.790331] Lustre: Skipped 1 previous similar message [ 716.808370] Lustre: Skipped 1 previous similar message [ 721.897487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 721.901843] Lustre: Skipped 1 previous similar message [ 725.104070] Lustre: server umount lustre-MDT0000 complete [ 727.009908] LustreError: 24259:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 727.024347] LustreError: 24259:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 728.031913] Lustre: 23574:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788439940/real 1788439940] req@00000000c3e74534 x1875315110475392/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788439956 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 728.059797] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 728.482080] LustreError: 24244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439957 with bad export cookie 15721568698502401294 [ 728.497528] LustreError: 24244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 728.499251] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 728.787911] Lustre: server umount lustre-MDT0001 complete [ 743.050900] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 745.680740] Key type lgssc unregistered [ 746.164926] LNet: 28236:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 746.177033] LNetError: 28236:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 746.190622] LNet: Removed LNI 192.168.203.122@tcp [ 747.577054] Key type .llcrypt unregistered [ 747.582085] Key type ._llcrypt unregistered [ 775.099648] Key type ._llcrypt registered [ 775.102219] Key type .llcrypt registered [ 775.203921] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 776.858178] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 776.913123] alg: No test for adler32 (adler32-zlib) [ 778.121386] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 778.500429] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 780.295158] Key type lgssc registered [ 781.557268] Lustre: Echo OBD driver; http://www.lustre.org/ [ 789.901984] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 789.918285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 791.426105] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 795.730716] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 801.771803] LustreError: 29648:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 801.781445] LustreError: 29648:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 804.972406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 805.571112] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 810.507220] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 818.377747] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 823.962438] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 836.821573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 837.192062] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 838.194351] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 843.411894] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 850.805365] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 858.592720] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 858.611815] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 858.624869] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 860.643294] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 860.652649] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 860.653453] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 863.686808] Lustre: server umount lustre-OST0000 complete [ 870.880685] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 870.903591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 870.906724] Lustre: Skipped 1 previous similar message [ 873.766448] Lustre: server umount lustre-MDT0000 complete [ 876.000922] LustreError: 30375:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 876.020114] LustreError: 30375:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 877.629625] LustreError: 32573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440106 with bad export cookie 15037949426952856416 [ 877.633529] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 877.637330] LustreError: 32573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 878.266725] Lustre: server umount lustre-MDT0001 complete [ 888.902730] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 891.629279] Key type lgssc unregistered [ 891.911573] LNet: 33524:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 891.917277] LNetError: 33524:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 891.936663] LNet: Removed LNI 192.168.203.122@tcp [ 892.836929] Key type .llcrypt unregistered [ 892.839055] Key type ._llcrypt unregistered [ 919.359731] Key type ._llcrypt registered [ 919.372668] Key type .llcrypt registered [ 919.552334] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 920.946584] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 921.025307] alg: No test for adler32 (adler32-zlib) [ 922.255155] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 922.429531] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 924.135539] Key type lgssc registered [ 925.289215] Lustre: Echo OBD driver; http://www.lustre.org/ [ 932.669215] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 932.698597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 934.115858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 938.383802] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 943.588376] LustreError: 34937:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 943.623166] LustreError: 34937:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 947.990697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 948.514107] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 953.308084] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 961.016701] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 966.806583] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 973.354695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 973.683118] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 978.817039] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 980.266274] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 988.524256] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1001.960224] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 1001.967529] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1001.984735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1003.488216] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1003.488959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1003.491682] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1003.516690] Lustre: Skipped 1 previous similar message [ 1008.633475] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1008.638954] Lustre: Skipped 1 previous similar message [ 1013.731825] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1013.746330] Lustre: Skipped 1 previous similar message [ 1014.239122] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1014.567432] Lustre: server umount lustre-OST0000 complete [ 1018.849280] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1018.862063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1018.863164] Lustre: Skipped 1 previous similar message [ 1018.885852] Lustre: Skipped 1 previous similar message [ 1029.093739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1029.104688] Lustre: Skipped 3 previous similar messages [ 1033.183143] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1033.459433] Lustre: server umount lustre-MDT0000 complete [ 1034.211393] LustreError: 34932:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1034.244380] LustreError: 34932:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1037.424973] LustreError: 34918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440266 with bad export cookie 5360277712074174726 [ 1037.432683] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1037.438265] LustreError: 34918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1037.740769] Lustre: server umount lustre-MDT0001 complete [ 1049.029300] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 1052.085296] Key type lgssc unregistered [ 1052.436174] LNet: 38472:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1052.460754] LNetError: 38472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1052.490546] LNet: Removed LNI 192.168.203.122@tcp [ 1053.377508] Key type .llcrypt unregistered [ 1053.380219] Key type ._llcrypt unregistered [ 1078.871490] Key type ._llcrypt registered [ 1078.874323] Key type .llcrypt registered [ 1079.015469] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 1080.628469] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1080.682523] alg: No test for adler32 (adler32-zlib) [ 1081.799322] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1082.073851] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 1083.783753] Key type lgssc registered [ 1085.184258] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1093.773312] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1093.800639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1095.352118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1100.597825] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1105.889768] LustreError: 39882:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1105.913579] LustreError: 39882:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1109.925384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1110.270851] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1114.797653] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1122.531468] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1128.189552] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1135.229512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1135.538076] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1140.671226] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1142.118386] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1150.236679] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1157.091987] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1157.102433] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1157.130212] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1158.316956] Lustre: lustre-OST0000: Not available for connect from 192.168.203.22@tcp (stopping) [ 1161.185661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1161.191097] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1161.193244] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1161.200480] Lustre: Skipped 1 previous similar message [ 1163.337804] Lustre: server umount lustre-OST0000 complete [ 1186.785158] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1186.791921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1186.799067] Lustre: Skipped 1 previous similar message [ 1186.813131] Lustre: Skipped 1 previous similar message [ 1189.337571] Lustre: server umount lustre-MDT0000 complete [ 1192.899395] LustreError: 39864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440421 with bad export cookie 6450322915965051277 [ 1192.904086] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1192.915817] LustreError: 39864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1207.263122] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1207.645646] Lustre: server umount lustre-MDT0001 complete [ 1218.449904] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 1221.110607] Key type lgssc unregistered [ 1221.417774] LNet: 43590:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1221.424436] LNetError: 43590:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1221.437511] LNet: Removed LNI 192.168.203.122@tcp [ 1222.475142] Key type .llcrypt unregistered [ 1222.477182] Key type ._llcrypt unregistered [ 1246.990350] Key type ._llcrypt registered [ 1246.993502] Key type .llcrypt registered [ 1247.112958] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 1248.313479] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1248.397507] alg: No test for adler32 (adler32-zlib) [ 1249.521300] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1249.794742] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 1251.511412] Key type lgssc registered [ 1252.935131] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1261.440500] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1261.460705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1262.992438] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1267.174198] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1272.291558] LustreError: 45013:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1272.317721] LustreError: 45013:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1275.481165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1275.861329] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1280.183399] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1287.845730] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1295.025061] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1302.570349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1303.044212] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1307.186160] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1310.433181] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1318.275780] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1331.363749] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.22@tcp (stopping) [ 1331.688494] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1331.704940] Lustre: Skipped 2 previous similar messages [ 1334.587728] Lustre: server umount lustre-MDT0000 complete [ 1336.445659] LustreError: 45010:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1338.773679] LustreError: 44994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440567 with bad export cookie 11102625523448344563 [ 1338.783357] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1338.788624] LustreError: 44994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1341.561387] LustreError: 45010:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1341.582371] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.22@tcp (stopping) [ 1341.593992] LustreError: 45010:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1341.616428] Lustre: Skipped 3 previous similar messages [ 1341.925711] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1343.968948] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1343.974056] LustreError: 45009:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1343.976815] Lustre: Skipped 1 previous similar message [ 1344.019544] LustreError: 45009:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1346.682445] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.22@tcp (stopping) [ 1349.089864] LustreError: 45010:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1349.112224] LustreError: 45010:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1351.802121] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.22@tcp (stopping) [ 1351.819063] Lustre: Skipped 1 previous similar message [ 1353.183159] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1353.462328] Lustre: server umount lustre-MDT0001 complete [ 1368.381579] Lustre: server umount lustre-OST0000 complete [ 1385.097402] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 1387.786400] Key type lgssc unregistered [ 1388.185230] LNet: 48873:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1388.190432] LNetError: 48873:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1388.210709] LNet: Removed LNI 192.168.203.122@tcp [ 1389.195584] Key type .llcrypt unregistered [ 1389.197682] Key type ._llcrypt unregistered [ 1415.470561] Key type ._llcrypt registered [ 1415.473248] Key type .llcrypt registered [ 1415.606369] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 1416.811548] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1416.869732] alg: No test for adler32 (adler32-zlib) [ 1418.049955] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1418.423731] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 1420.183220] Key type lgssc registered [ 1421.633739] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1430.935550] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1430.977818] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1446.560312] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1451.680038] LustreError: 50257:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.122@tcp: failed processing log, type 4: rc = -110 [ 1478.367185] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1484.559572] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1491.841538] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1656.902919] Lustre: server umount lustre-OST0000 complete [ 1671.996987] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 1674.176089] Key type lgssc unregistered [ 1674.397774] LNet: 51932:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1674.402185] LNetError: 51932:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1674.421331] LNet: Removed LNI 192.168.203.122@tcp [ 1674.993466] Key type .llcrypt unregistered [ 1674.996039] Key type ._llcrypt unregistered [ 1699.040793] Key type ._llcrypt registered [ 1699.045271] Key type .llcrypt registered [ 1699.146512] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 1700.146203] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1700.189402] alg: No test for adler32 (adler32-zlib) [ 1701.234721] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1701.409855] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 1703.055324] Key type lgssc registered [ 1704.189615] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1712.253488] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1712.269276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1713.632981] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1717.941899] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1723.878815] LustreError: 53342:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1723.922945] LustreError: 53342:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1726.885761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1727.386909] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1732.450594] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1740.292351] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1746.255802] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1753.497179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1753.876227] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1759.038584] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1760.425514] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1767.768066] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1772.901388] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1772.901476] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1778.149929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1778.155795] Lustre: Skipped 1 previous similar message [ 1783.266172] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1783.276206] Lustre: Skipped 1 previous similar message [ 1784.287605] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1784.472833] Lustre: server umount lustre-OST0000 complete [ 1788.384926] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1788.388198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1788.399300] Lustre: Skipped 1 previous similar message [ 1793.507892] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1793.517729] Lustre: Skipped 2 previous similar messages [ 1794.365669] Lustre: server umount lustre-MDT0000 complete [ 1797.974937] LustreError: 54077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441026 with bad export cookie 12334856091710674879 [ 1797.975564] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1797.998930] LustreError: 54077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1798.446731] Lustre: server umount lustre-MDT0001 complete [ 1809.544459] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 1812.523522] Key type lgssc unregistered [ 1812.821562] LNet: 56874:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1812.837340] LNetError: 56874:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1812.862380] LNet: Removed LNI 192.168.203.122@tcp [ 1813.845187] Key type .llcrypt unregistered [ 1813.848281] Key type ._llcrypt unregistered [ 1834.311047] Key type ._llcrypt registered [ 1834.312655] Key type .llcrypt registered [ 1834.406823] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 1835.361035] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1835.508626] alg: No test for adler32 (adler32-zlib) [ 1836.573473] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1836.786950] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 1838.442449] Key type lgssc registered [ 1839.437842] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1846.855400] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1846.884849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1862.431576] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1867.551393] LustreError: 58236:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.122@tcp: failed processing log, type 4: rc = -110 [ 1894.239934] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1901.346814] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1910.806304] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1917.778591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1918.175770] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1918.229495] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1922.345986] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1923.557118] LustreError: 58261:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1923.586262] LustreError: 58261:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1928.689965] LustreError: 58262:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1928.716524] LustreError: 58262:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1930.842538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1931.418105] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1935.955299] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1942.863518] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1948.420785] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1951.713630] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1951.727050] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1951.764174] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1956.347558] Lustre: server umount lustre-OST0000 complete [ 1980.499947] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1980.520299] Lustre: Skipped 1 previous similar message [ 1980.525551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1980.536265] Lustre: Skipped 3 previous similar messages [ 1982.502424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1982.511518] Lustre: Skipped 1 previous similar message [ 1986.799115] Lustre: server umount lustre-MDT0000 complete [ 1987.569223] LustreError: 59277:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1987.598462] LustreError: 59277:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1990.627378] LustreError: 58242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441219 with bad export cookie 17946901273008107318 [ 1990.630930] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1990.643389] LustreError: 58242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1997.158292] Lustre: server umount lustre-MDT0001 complete [ 2012.001230] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 2015.577726] Key type lgssc unregistered [ 2016.054166] LNet: 61986:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2016.068195] LNetError: 61986:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2016.116659] LNet: Removed LNI 192.168.203.122@tcp [ 2017.487062] Key type .llcrypt unregistered [ 2017.491851] Key type ._llcrypt unregistered [ 2042.083529] Key type ._llcrypt registered [ 2042.085879] Key type .llcrypt registered [ 2042.243985] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 2043.496905] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2043.669094] alg: No test for adler32 (adler32-zlib) [ 2044.766565] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2045.116645] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 2046.817145] Key type lgssc registered [ 2047.855645] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2055.810891] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2055.854604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2057.424508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2061.973576] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2067.937134] LustreError: 63401:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2067.966267] LustreError: 63401:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2070.098028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2070.608253] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2074.949938] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2081.978616] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2087.489938] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2093.903130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2094.276941] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2100.197351] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2102.450195] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2107.086638] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2125.300601] Lustre: server umount lustre-OST0000 complete [ 2131.936310] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2131.945460] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2131.961230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2133.474118] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2133.474872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2133.510905] Lustre: Skipped 1 previous similar message [ 2138.592313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2138.600687] Lustre: Skipped 1 previous similar message [ 2139.705452] Lustre: server umount lustre-MDT0000 complete [ 2142.303137] Lustre: 62707:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788441355/real 1788441355] req@000000007f11b146 x1875316597355648/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788441371 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2142.325832] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2142.346922] LustreError: 63401:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2142.365780] LustreError: 63401:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2143.412469] LustreError: 63382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441372 with bad export cookie 4841062945743436837 [ 2143.435650] LustreError: 63382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2143.435860] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2143.809589] Lustre: server umount lustre-MDT0001 complete [ 2154.206370] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 2156.880244] Key type lgssc unregistered [ 2157.164316] LNet: 66958:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2157.180053] LNetError: 66958:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2157.207445] LNet: Removed LNI 192.168.203.122@tcp [ 2158.081082] Key type .llcrypt unregistered [ 2158.084626] Key type ._llcrypt unregistered [ 2182.010077] Key type ._llcrypt registered [ 2182.012238] Key type .llcrypt registered [ 2182.091536] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 2182.932163] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2182.980989] alg: No test for adler32 (adler32-zlib) [ 2184.104837] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2184.372483] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 2186.031164] Key type lgssc registered [ 2187.030603] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2195.761232] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 09:17:02 (1788441422) [ 2203.229868] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2203.243378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2204.683222] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2209.597232] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2214.884657] LustreError: 68660:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2214.903605] LustreError: 68660:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2217.932699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2218.206456] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2222.651820] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2230.016687] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2235.658127] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2241.731024] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2242.097914] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2248.059720] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2249.272425] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2255.118715] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2261.516495] Lustre: Failing over lustre-MDT0000 [ 2261.731588] Lustre: server umount lustre-MDT0000 complete [ 2263.201947] LustreError: 68655:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2264.550720] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2264.557330] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2264.562190] LustreError: 68660:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2264.567300] Lustre: Skipped 1 previous similar message [ 2268.283186] LustreError: 68654:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2268.313886] LustreError: 68654:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2270.826240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2270.965189] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2271.169840] Lustre: *** cfs_fail_loc=135, val=0*** [ 2271.284860] LustreError: 71748:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2271.299839] LustreError: 71748:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.122@tcp: cfg command failed: rc = -2 [ 2271.313339] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2271.313339] [ 2271.326301] LustreError: MGC192.168.203.122@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2271.338586] LustreError: 71736:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2271.347658] LustreError: 71736:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [ 2271.370250] LustreError: 71736:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 2271.377279] LustreError: 71736:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2271.405820] Lustre: server umount lustre-MDT0000 complete [ 2271.408477] LustreError: 71736:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2273.405503] LustreError: 68654:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2273.430014] LustreError: 68654:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2277.606321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2277.675913] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2277.940370] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2278.047573] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2278.528073] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2282.876430] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2282.991033] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2283.053027] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2283.128426] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2293.221881] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2293.228571] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2293.240437] Lustre: Skipped 2 previous similar messages [ 2293.255728] Lustre: Skipped 1 previous similar message [ 2297.221609] Lustre: server umount lustre-OST0000 complete [ 2303.456312] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2303.465197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2303.473455] Lustre: Skipped 1 previous similar message [ 2303.482916] Lustre: Skipped 1 previous similar message [ 2307.538531] Lustre: server umount lustre-MDT0000 complete [ 2308.584359] LustreError: 68655:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2308.606111] LustreError: 68655:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2311.441993] LustreError: 68640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441540 with bad export cookie 3343151042014004078 [ 2311.443700] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2311.459676] LustreError: 68640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2311.815490] Lustre: server umount lustre-MDT0001 complete [ 2322.821717] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 2325.357355] Key type lgssc unregistered [ 2325.648626] LNet: 73524:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2325.660350] LNetError: 73524:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2326.694110] LNet: Removed LNI 192.168.203.122@tcp [ 2327.669448] Key type .llcrypt unregistered [ 2327.673819] Key type ._llcrypt unregistered [ 2350.409656] Key type ._llcrypt registered [ 2350.411527] Key type .llcrypt registered [ 2350.507957] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 2351.564750] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2351.608036] alg: No test for adler32 (adler32-zlib) [ 2352.778773] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2353.080047] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 2354.759182] Key type lgssc registered [ 2355.763345] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2363.408789] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2363.421309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2364.964620] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2368.270866] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2374.115653] LustreError: 74931:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2374.125052] LustreError: 74931:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2375.487456] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2375.760232] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2380.469376] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2389.677073] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2395.923830] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2403.172302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2403.557533] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2409.441807] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2409.745548] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2418.076391] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2427.647239] Lustre: Failing over lustre-MDT0000 [ 2428.379899] Lustre: server umount lustre-MDT0000 complete [ 2430.116862] LustreError: 74926:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2431.464084] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2431.498713] LustreError: 74927:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2431.530158] Lustre: Skipped 1 previous similar message [ 2435.206196] LustreError: 74928:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2435.223708] LustreError: 74928:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2437.903745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2438.044908] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2438.148309] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2438.266086] LustreError: 78019:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2438.274343] LustreError: 78019:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.122@tcp: cfg command failed: rc = -9 [ 2438.287367] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2438.287367] [ 2438.302902] LustreError: MGC192.168.203.122@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2438.314912] LustreError: 78007:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2438.326362] LustreError: 78007:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -9 [ 2438.333550] LustreError: 78007:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 2438.344505] LustreError: 78007:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2438.372218] Lustre: server umount lustre-MDT0000 complete [ 2438.376442] LustreError: 78007:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2440.313244] LustreError: 77058:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2440.338098] LustreError: 77058:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2443.934961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2444.131929] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2444.624693] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2444.739541] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2445.434070] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2449.272575] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2449.896173] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2449.918832] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2449.970834] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2455.523833] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2455.536636] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2455.558155] Lustre: Skipped 1 previous similar message [ 2455.570309] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2457.056333] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2457.061493] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2457.079990] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2457.555429] Lustre: server umount lustre-OST0000 complete [ 2465.250596] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2465.266570] Lustre: Skipped 1 previous similar message [ 2465.269554] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2465.269751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2466.795661] Lustre: server umount lustre-MDT0000 complete [ 2469.891897] LustreError: 74914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441698 with bad export cookie 11694624061972417738 [ 2469.895491] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2469.900624] LustreError: 74914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2470.239916] Lustre: server umount lustre-MDT0001 complete [ 2480.228787] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 2483.192530] Key type lgssc unregistered [ 2483.486404] LNet: 79797:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2483.501274] LNetError: 79797:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2483.522667] LNet: Removed LNI 192.168.203.122@tcp [ 2484.380364] Key type .llcrypt unregistered [ 2484.383973] Key type ._llcrypt unregistered [ 2498.625183] Key type ._llcrypt registered [ 2498.639021] Key type .llcrypt registered [ 2498.818253] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 2499.263012] Key type .llcrypt unregistered [ 2499.271518] Key type ._llcrypt unregistered [ 2523.641550] Key type ._llcrypt registered [ 2523.644089] Key type .llcrypt registered [ 2523.728194] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 2524.867164] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2524.908415] alg: No test for adler32 (adler32-zlib) [ 2525.899247] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2526.092198] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 2527.807215] Key type lgssc registered [ 2529.084395] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2537.645977] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2537.670787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2539.270104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2544.100941] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2549.729974] LustreError: 81695:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2549.743834] LustreError: 81695:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2552.516971] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2552.858036] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2557.466444] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2565.293105] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2571.672994] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2578.965735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2579.441963] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2584.568524] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2586.053204] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2595.815476] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2610.152420] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2610.166375] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2610.186783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2613.216847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2613.223178] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2613.225077] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2615.474051] Lustre: server umount lustre-OST0000 complete [ 2623.456888] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2623.479797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2623.484293] Lustre: Skipped 2 previous similar messages [ 2626.479888] Lustre: server umount lustre-MDT0000 complete [ 2628.579782] LustreError: 83993:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2628.620289] LustreError: 83993:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2630.810026] LustreError: 81674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441859 with bad export cookie 15910300884289227468 [ 2630.816931] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2630.820759] LustreError: 81674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2631.335178] Lustre: server umount lustre-MDT0001 complete [ 2642.727678] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 2645.539201] Key type lgssc unregistered [ 2645.791614] LNet: 85249:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2645.804582] LNetError: 85249:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2645.838874] LNet: Removed LNI 192.168.203.122@tcp [ 2646.603943] Key type .llcrypt unregistered [ 2646.606237] Key type ._llcrypt unregistered [ 2671.205231] Key type ._llcrypt registered [ 2671.206933] Key type .llcrypt registered [ 2671.373641] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 2672.688522] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2673.044816] alg: No test for adler32 (adler32-zlib) [ 2674.297354] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2674.636474] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 2676.295168] Key type lgssc registered [ 2677.453575] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2685.751359] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2685.778933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2687.197073] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2691.573518] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2697.713457] LustreError: 86660:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2697.752697] LustreError: 86660:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2700.217819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2700.751424] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2705.458321] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2712.847837] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2718.629982] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2725.566181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2725.909880] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2732.112219] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2733.124654] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2739.871093] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2750.435594] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2750.441056] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2750.451019] Lustre: Skipped 1 previous similar message [ 2750.472233] Lustre: Skipped 1 previous similar message [ 2751.698467] Lustre: server umount lustre-OST0000 complete [ 2757.090677] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2757.094253] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2757.113027] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2760.672698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2760.672704] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2760.702976] Lustre: Skipped 1 previous similar message [ 2762.200717] Lustre: server umount lustre-MDT0000 complete [ 2765.794985] LustreError: 86659:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2765.809676] LustreError: 86659:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2766.046523] LustreError: 86641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441994 with bad export cookie 11447206143050304968 [ 2766.061865] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2766.064365] LustreError: 86641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2766.325818] Lustre: server umount lustre-MDT0001 complete [ 2777.328495] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 2780.143232] Key type lgssc unregistered [ 2780.530259] LNet: 90189:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2780.535279] LNetError: 90189:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2780.548410] LNet: Removed LNI 192.168.203.122@tcp [ 2781.454554] Key type .llcrypt unregistered [ 2781.457601] Key type ._llcrypt unregistered [ 2805.330950] Key type ._llcrypt registered [ 2805.334724] Key type .llcrypt registered [ 2805.453740] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 2806.632756] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2806.750853] alg: No test for adler32 (adler32-zlib) [ 2807.817694] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2808.121116] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 2809.831787] Key type lgssc registered [ 2811.294622] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2819.590594] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2819.619868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2821.216114] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2825.776656] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2831.332407] LustreError: 91599:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2831.365952] LustreError: 91599:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2834.757796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2835.181869] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2840.855452] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2849.159231] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2855.455112] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2862.569680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2863.010687] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2867.133170] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2869.302818] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2876.556688] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2892.772470] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2892.788176] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2892.809673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2893.378138] Lustre: server umount lustre-OST0000 complete [ 2899.395172] Lustre: server umount lustre-MDT0000 complete [ 2900.962174] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2900.966191] LustreError: 91598:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2900.984522] Lustre: Skipped 1 previous similar message [ 2902.886159] LustreError: 91579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442131 with bad export cookie 9613498870724310019 [ 2902.898104] LustreError: 91579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2902.899721] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2912.371410] Lustre: server umount lustre-MDT0001 complete [ 2922.662978] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 2925.410901] Key type lgssc unregistered [ 2925.719344] LNet: 95153:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2925.728904] LNetError: 95153:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2925.757840] LNet: Removed LNI 192.168.203.122@tcp [ 2926.550266] Key type .llcrypt unregistered [ 2926.552390] Key type ._llcrypt unregistered [ 2950.242898] Key type ._llcrypt registered [ 2950.245136] Key type .llcrypt registered [ 2950.360202] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 2951.645905] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2951.690881] alg: No test for adler32 (adler32-zlib) [ 2952.772022] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2953.114472] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 2954.815223] Key type lgssc registered [ 2956.117574] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2964.267937] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2964.316261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2979.935518] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2985.058614] LustreError: 96539:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.122@tcp: failed processing log, type 4: rc = -110 [ 3011.743216] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3018.886534] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3027.057549] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3032.702158] Lustre: server umount lustre-OST0000 complete [ 3041.376526] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 09:31:08 (1788442268) [ 3048.443279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3048.820038] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3053.807272] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3054.065819] LustreError: 98179:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3054.078109] LustreError: 98179:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3058.150987] LustreError: 98180:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3062.895353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3063.367247] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3068.129340] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3075.629724] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3081.634147] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3087.822938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3088.077731] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3093.234338] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3094.569512] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3102.173813] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3114.978811] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3114.995095] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3115.017761] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3119.075759] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3119.075994] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3119.080293] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3120.910189] Lustre: server umount lustre-OST0000 complete [ 3124.192966] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3124.198951] Lustre: Skipped 1 previous similar message [ 3124.199722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3124.212763] Lustre: Skipped 2 previous similar messages [ 3129.311816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3129.326308] Lustre: Skipped 1 previous similar message [ 3130.248125] Lustre: server umount lustre-MDT0000 complete [ 3133.833216] LustreError: 98161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442362 with bad export cookie 12049470558809780342 [ 3133.836735] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3133.842153] LustreError: 98161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3134.238713] Lustre: server umount lustre-MDT0001 complete [ 3144.936721] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 3147.725747] Key type lgssc unregistered [ 3148.076639] LNet: 101869:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3148.091506] LNetError: 101869:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3148.130268] LNet: Removed LNI 192.168.203.122@tcp [ 3149.102039] Key type .llcrypt unregistered [ 3149.105551] Key type ._llcrypt unregistered [ 3190.853273] Key type ._llcrypt registered [ 3190.857447] Key type .llcrypt registered [ 3190.952567] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 3192.051304] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3192.078621] alg: No test for adler32 (adler32-zlib) [ 3193.225425] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3193.546228] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 3195.225485] Key type lgssc registered [ 3196.636341] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3204.613191] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3204.625808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3206.186557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3210.804502] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3216.357304] LustreError: 104053:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3216.373636] LustreError: 104053:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3219.505456] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3219.840663] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3224.350109] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3232.525863] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3238.680798] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3246.326404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3246.707328] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3251.839421] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3253.541201] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3262.103151] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3275.744541] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3275.753090] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3275.765272] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3275.806245] Lustre: Skipped 1 previous similar message [ 3280.875436] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3280.895211] Lustre: Skipped 1 previous similar message [ 3281.106985] Lustre: server umount lustre-OST0000 complete [ 3285.990134] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3285.996469] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3286.012298] Lustre: Skipped 1 previous similar message [ 3286.029356] Lustre: Skipped 2 previous similar messages [ 3291.109044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3291.503384] Lustre: server umount lustre-MDT0000 complete [ 3295.488118] LustreError: 104035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442524 with bad export cookie 6116180972234626730 [ 3295.500251] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3295.509254] LustreError: 104035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3295.948808] Lustre: server umount lustre-MDT0001 complete [ 3308.662895] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 3312.098073] Key type lgssc unregistered [ 3312.394652] LNet: 107608:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3312.400636] LNetError: 107608:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3312.422612] LNet: Removed LNI 192.168.203.122@tcp [ 3313.246892] Key type .llcrypt unregistered [ 3313.250516] Key type ._llcrypt unregistered [ 3332.447947] Key type ._llcrypt registered [ 3332.457623] Key type .llcrypt registered [ 3332.600177] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 3333.768942] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3333.970926] alg: No test for adler32 (adler32-zlib) [ 3335.122445] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3335.416797] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 3337.095145] Key type lgssc registered [ 3338.240799] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3346.383527] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3346.397695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3362.016257] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3367.135299] LustreError: 108848:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.122@tcp: failed processing log, type 4: rc = -110 [ 3393.823537] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3401.783075] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3409.417626] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3415.491218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3415.563511] LustreError: 108183:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000098576007 x1875317949991168/t0(0) o250->MGC192.168.203.122@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3415.763039] LustreError: MGC192.168.203.122@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3415.776564] LustreError: 109870:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3415.782985] LustreError: 109870:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [ 3415.793777] LustreError: 109870:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 3415.852419] Lustre: server umount lustre-MDT0000 complete [ 3415.855423] LustreError: 109870:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3437.023169] Lustre: 108185:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788442649/real 1788442649] req@0000000001731c20 x1875317949992320/t0(0) o400->MGC192.168.203.122@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788442665 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3437.052761] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3437.062138] LustreError: 108860:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3437.208377] Lustre: server umount lustre-OST0000 complete [ 3446.584834] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 3459.646539] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 3497.563706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3497.797188] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3497.827864] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3497.919311] Lustre: lustre-MDT0000: new disk, initializing [ 3498.034163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3498.047372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3502.382633] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3512.744427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3512.851159] Lustre: 112663:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3512.896839] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3512.900908] Lustre: Skipped 1 previous similar message [ 3513.051576] Lustre: lustre-MDT0001: new disk, initializing [ 3513.179558] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3513.224648] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3513.241534] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3517.697739] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3526.711497] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3532.043573] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3538.936345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3539.239304] Lustre: lustre-OST0000: new disk, initializing [ 3539.247487] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3539.251991] Lustre: 114631:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3539.363777] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3545.218315] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3547.208463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3547.221981] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3547.315674] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3554.736548] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3559.592935] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3559.803181] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3563.832149] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3564.073543] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3568.097091] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3568.104379] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3568.123268] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3568.134466] Lustre: Skipped 1 previous similar message [ 3573.220833] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3573.229591] Lustre: Skipped 1 previous similar message [ 3578.340738] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3578.347790] Lustre: Skipped 1 previous similar message [ 3580.383111] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3580.557705] Lustre: server umount lustre-OST0000 complete [ 3584.995434] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3585.004384] LustreError: Skipped 1 previous similar message [ 3585.011273] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3585.023495] Lustre: Skipped 1 previous similar message [ 3585.028289] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3588.579640] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3590.932927] Lustre: server umount lustre-MDT0000 complete [ 3594.560248] LustreError: 112653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442823 with bad export cookie 1421941919285006038 [ 3594.576526] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3594.590121] LustreError: 112653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3594.943749] Lustre: server umount lustre-MDT0001 complete [ 3603.228359] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 09:40:30 (1788442830) [ 3606.759400] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3622.661076] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 3632.059458] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 3668.137544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3668.272215] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3668.310475] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3668.354791] Lustre: lustre-MDT0000: new disk, initializing [ 3668.433232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3668.447582] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3672.890521] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3682.628429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3682.781430] Lustre: 119535:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3682.898637] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3682.905120] Lustre: Skipped 1 previous similar message [ 3682.978221] Lustre: lustre-MDT0001: new disk, initializing [ 3683.087238] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3683.127313] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3683.205702] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3689.727456] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3702.571445] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3707.647442] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3714.050295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3714.308700] Lustre: lustre-OST0000: new disk, initializing [ 3714.312756] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3714.316744] Lustre: 121503:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3714.370376] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3720.241533] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3722.816815] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3722.830151] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3722.944742] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3729.121972] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3734.095694] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3734.334682] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3739.120786] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3739.403509] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3743.718491] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3743.751511] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3743.761153] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3743.773598] Lustre: Skipped 3 previous similar messages [ 3747.619217] Lustre: server umount lustre-OST0000 complete [ 3753.956249] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3753.976426] Lustre: Skipped 1 previous similar message [ 3753.990501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3758.820321] Lustre: server umount lustre-MDT0000 complete [ 3759.073902] LustreError: 119542:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3759.084499] LustreError: 119542:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3761.964884] LustreError: 119527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442990 with bad export cookie 1421941919285006913 [ 3761.968688] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3761.982518] LustreError: 119527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3762.487256] Lustre: server umount lustre-MDT0001 complete [ 3770.976280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3771.380629] LustreError: 123718:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3771.440629] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3775.420859] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3776.483836] LustreError: 123719:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3781.609946] LustreError: 123718:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3785.796934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3791.088269] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3799.481773] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3805.040938] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3811.756279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3819.107017] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3827.248894] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3837.145385] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3842.028843] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3842.036228] Lustre: Skipped 1 previous similar message [ 3842.046672] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3842.053068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3842.070224] Lustre: Skipped 1 previous similar message [ 3847.152491] LustreError: 125537:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3847.163318] LustreError: 125537:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3847.334484] Lustre: server umount lustre-OST0000 complete [ 3857.382881] LustreError: 123718:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3857.405114] LustreError: 123718:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3857.543399] Lustre: server umount lustre-MDT0000 complete [ 3861.780990] LustreError: 123699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443090 with bad export cookie 1421941919285007802 [ 3861.800956] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3861.823398] LustreError: 123699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3862.560605] Lustre: server umount lustre-MDT0001 complete [ 3873.994861] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 3876.885617] Key type lgssc unregistered [ 3877.193719] LNet: 127347:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3877.206252] LNetError: 127347:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3877.228276] LNet: Removed LNI 192.168.203.122@tcp [ 3878.170196] Key type .llcrypt unregistered [ 3878.172446] Key type ._llcrypt unregistered [ 3895.610729] Key type ._llcrypt registered [ 3895.615296] Key type .llcrypt registered [ 3895.711079] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 3911.003915] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 3912.119358] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3912.149090] alg: No test for adler32 (adler32-zlib) [ 3913.320731] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3913.668951] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 3915.487224] Key type lgssc registered [ 3916.567164] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3954.437571] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3954.452963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3955.633811] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3955.677636] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3955.743602] Lustre: lustre-MDT0000: new disk, initializing [ 3955.928454] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3955.960863] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3960.210169] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3970.935771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3971.155109] Lustre: 130499:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3971.281079] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3971.299207] Lustre: Skipped 1 previous similar message [ 3971.446237] Lustre: lustre-MDT0001: new disk, initializing [ 3971.635257] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3971.719277] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3971.743935] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3976.705650] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3986.027488] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3991.875804] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3998.496915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3998.766778] Lustre: lustre-OST0000: new disk, initializing [ 3998.773297] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3998.783142] Lustre: 132468:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3998.855227] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4004.401961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4004.408850] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4004.535043] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4005.152508] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4013.828079] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4018.608763] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4018.824496] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4023.417989] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4023.630866] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4030.441168] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4030.442904] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4030.483863] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4031.660207] Lustre: server umount lustre-OST0000 complete [ 4035.554758] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4035.556289] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4035.573643] Lustre: Skipped 2 previous similar messages [ 4035.587618] Lustre: Skipped 2 previous similar messages [ 4040.675893] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4040.690135] Lustre: Skipped 1 previous similar message [ 4041.770908] Lustre: server umount lustre-MDT0000 complete [ 4045.397866] LustreError: 130489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443274 with bad export cookie 2523322857952626267 [ 4045.398289] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4045.407416] LustreError: 130489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4045.779299] Lustre: server umount lustre-MDT0001 complete [ 4053.302391] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 09:48:00 (1788443280) [ 4059.327968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4059.890420] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4063.909833] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4065.254279] LustreError: 134878:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4065.275523] LustreError: 134878:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4069.350249] LustreError: 134878:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4072.333879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4072.824718] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4077.594366] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4085.122137] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4091.498406] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4094.433069] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4094.436072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4094.452270] Lustre: Skipped 1 previous similar message [ 4094.468676] Lustre: Skipped 1 previous similar message [ 4100.080119] Lustre: server umount lustre-MDT0000 complete [ 4102.625881] LustreError: 135607:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4102.649955] LustreError: 135607:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4103.839570] LustreError: 134858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443332 with bad export cookie 2523322857952627149 [ 4103.845168] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4103.852267] LustreError: 134858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4104.233688] Lustre: server umount lustre-MDT0001 complete [ 4112.408642] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 09:48:59 (1788443339) [ 4118.404549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4134.113700] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4139.231974] LustreError: 137293:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.122@tcp: failed processing log, type 4: rc = -110 [ 4164.973263] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4171.042912] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4178.581663] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4180.574308] Lustre: server umount lustre-OST0000 complete [ 4188.285517] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 09:50:15 (1788443415) [ 4194.149782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4194.526974] LustreError: 138740:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4194.602480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4198.368718] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4199.906468] LustreError: 138741:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4206.026052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4210.221817] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4217.545276] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4223.032990] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4228.468986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4228.718506] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4228.727332] Lustre: Skipped 1 previous similar message [ 4233.586963] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4239.936979] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4247.521085] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4247.536655] LustreError: Skipped 1 previous similar message [ 4247.539562] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4247.568265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4247.576076] Lustre: Skipped 2 previous similar messages [ 4253.493889] Lustre: server umount lustre-MDT0000 complete [ 4255.719030] LustreError: 139983:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4255.742332] LustreError: 139983:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4257.169632] LustreError: 138721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443486 with bad export cookie 2523322857952627814 [ 4257.175336] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4257.188601] LustreError: 138721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4257.502608] Lustre: server umount lustre-MDT0001 complete [ 4271.615886] Lustre: server umount lustre-OST0000 complete [ 4279.768897] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 09:51:46 (1788443506) [ 4284.997144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4285.330478] LustreError: 142394:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4289.573621] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4297.009202] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4297.336713] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4297.344427] Lustre: Skipped 1 previous similar message [ 4301.647133] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4309.368636] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4316.977595] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4325.415833] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4331.077061] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4332.358554] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4339.563375] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4343.882228] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4344.164813] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4349.013860] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4349.244706] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4352.501564] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4352.518410] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4352.540309] Lustre: Skipped 2 previous similar messages [ 4352.556652] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4352.559252] Lustre: Skipped 3 previous similar messages [ 4358.159842] Lustre: server umount lustre-OST0000 complete [ 4363.745069] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4363.763877] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4363.781755] Lustre: Skipped 1 previous similar message [ 4369.930292] Lustre: server umount lustre-MDT0000 complete [ 4372.960578] LustreError: 143462:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4372.972302] LustreError: 143462:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4373.485642] LustreError: 142374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443602 with bad export cookie 2523322857952628878 [ 4373.492113] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4373.505846] LustreError: 142374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4373.760926] Lustre: server umount lustre-MDT0001 complete [ 4380.933832] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 09:53:28 (1788443608) [ 4386.194984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4401.951939] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4407.072619] LustreError: 146552:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.122@tcp: failed processing log, type 4: rc = -110 [ 4432.867229] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4432.870753] Lustre: Skipped 1 previous similar message [ 4440.126789] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4449.334225] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4456.239264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4456.820501] LustreError: 146577:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4457.009294] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4462.942254] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4473.542944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4478.825333] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4487.499874] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4494.302804] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4500.405989] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4500.808493] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4505.110783] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4505.462193] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4509.664978] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4509.667816] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4509.671501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4509.678371] Lustre: Skipped 3 previous similar messages [ 4509.684849] Lustre: Skipped 7 previous similar messages [ 4513.660954] Lustre: server umount lustre-MDT0000 complete [ 4517.697926] LustreError: 146558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443746 with bad export cookie 2523322857952629907 [ 4517.715782] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4518.069252] Lustre: server umount lustre-MDT0001 complete [ 4532.343290] Lustre: server umount lustre-OST0000 complete [ 4544.122878] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 09:56:10 (1788443770) [ 4552.498324] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4568.159681] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4573.282434] LustreError: 150746:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.122@tcp: failed processing log, type 4: rc = -110 [ 4606.586270] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4614.904486] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4622.133630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4622.542344] LustreError: 150771:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4622.575967] LustreError: 150771:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 4622.703158] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4627.088939] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4635.449694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4640.782923] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4648.113906] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4653.719282] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4660.841588] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4661.122657] Lustre: lustre-OST0001: new disk, initializing [ 4661.126251] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4661.136682] Lustre: 153617:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4668.487412] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4669.020532] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4669.030466] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4669.170291] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4678.766644] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4683.501610] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4683.784858] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4688.385225] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4688.645119] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4692.450644] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4692.464667] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4692.486958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4697.038812] Lustre: server umount lustre-OST0000 complete [ 4706.855753] Lustre: server umount lustre-OST0001 complete [ 4712.930513] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4712.945734] LustreError: Skipped 3 previous similar messages [ 4717.917207] Lustre: server umount lustre-MDT0000 complete [ 4721.308470] LustreError: 150754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443950 with bad export cookie 2523322857952630859 [ 4721.309323] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4721.327684] LustreError: 150754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 4721.839577] Lustre: server umount lustre-MDT0001 complete [ 4751.979022] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 09:59:39 (1788443979) [ 4753.927378] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4755.805678] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4757.985300] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 09:59:44 (1788443984) [ 4765.435929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4765.894401] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4765.946440] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4766.338627] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4766.344704] Lustre: Skipped 6 previous similar messages [ 4771.114809] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4779.707313] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4779.758756] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4779.775276] Lustre: 157180:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4785.241572] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4794.622310] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4804.353575] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4815.649637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4815.877046] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4822.645605] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4823.163440] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4830.288574] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4835.436506] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4835.637816] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4840.827742] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4841.017817] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4843.488762] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4843.500598] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4843.521156] Lustre: Skipped 5 previous similar messages [ 4843.531109] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4843.538294] Lustre: Skipped 8 previous similar messages [ 4857.315383] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4857.455574] Lustre: server umount lustre-OST0000 complete [ 4882.484254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4884.925246] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4884.958315] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4884.964148] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4890.236350] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4901.765924] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4909.423717] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4909.715290] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4915.047711] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4915.284967] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4921.713189] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9ad890a08800.ost_server_uuid 50 [ 4923.436981] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9ad890a08800.ost_server_uuid in FULL state after 0 sec [ 4923.553770] LustreError: 157185:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.203.22@tcp) returned error from blocking AST (req@000000008e88a082 x1875318556337024 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 000000009460095e/0x2304a4c529e3e1a5 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.22@tcp remote: 0x1dec426e1c696366 expref: 5 pid: 157185 timeout: 5023 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4923.603337] LustreError: lustre-MDT0000: A client on nid 192.168.203.22@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4923.621909] LustreError: 157175:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.203.22@tcp ns: mdt-lustre-MDT0000_UUID lock: 000000009460095e/0x2304a4c529e3e1a5 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.22@tcp remote: 0x1dec426e1c696366 expref: 6 pid: 157185 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4928.995572] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4929.002090] LustreError: Skipped 1 previous similar message [ 4929.917885] Lustre: server umount lustre-OST0000 complete [ 4940.702454] Lustre: server umount lustre-MDT0000 complete [ 4941.291136] LustreError: 157190:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4941.308209] LustreError: 157190:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 4944.281304] LustreError: 157170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444173 with bad export cookie 2523322857952632042 [ 4944.293225] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4954.813555] Lustre: server umount lustre-MDT0001 complete [ 4969.532658] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 4973.068991] Key type lgssc unregistered [ 4973.402175] LNet: 163310:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4973.421783] LNetError: 163310:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4973.454320] LNet: Removed LNI 192.168.203.122@tcp [ 4974.385520] Key type .llcrypt unregistered [ 4974.388582] Key type ._llcrypt unregistered [ 4996.922793] Key type ._llcrypt registered [ 4996.927497] Key type .llcrypt registered [ 4997.024494] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 4998.403901] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4998.516100] alg: No test for adler32 (adler32-zlib) [ 4999.701900] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 4999.940410] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 5001.671159] Key type lgssc registered [ 5002.610365] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5010.569673] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5010.623272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5012.168453] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5017.000925] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5022.698450] LustreError: 164719:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5022.721474] LustreError: 164719:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5025.494351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5025.899422] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5030.102879] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5037.016376] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5042.294917] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5048.592286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5048.930775] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5055.161446] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5057.156738] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5063.283855] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5069.073117] Lustre: Failing over lustre-MDT0000 [ 5069.377273] Lustre: server umount lustre-MDT0000 complete [ 5069.991460] LustreError: 165775:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5071.841828] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5071.851707] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5071.868589] LustreError: 164719:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5072.865604] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5072.892620] Lustre: Skipped 1 previous similar message [ 5077.986287] LustreError: 164720:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5078.022458] LustreError: 164720:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 5081.513475] LustreError: 164699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.22@tcp arrived at 1788444310 with bad export cookie 3583217781047097566 [ 5083.106049] LustreError: 164719:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5083.119765] LustreError: 164719:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5088.229260] Lustre: 164029:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788444301/real 1788444301] req@00000000f1af8a33 x1875319694846336/t0(0) o400->MGC192.168.203.122@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788444317 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5088.289865] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5090.008298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5093.347797] LustreError: 164719:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5093.365535] LustreError: 164719:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 5098.475441] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x31ba2596502cf940 [ 5098.490783] Lustre: MGC192.168.203.122@tcp: Connection restored to 0@lo (at 0@lo) [ 5098.772719] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5098.820867] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5103.425696] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5104.106649] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5104.113215] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5105.856517] Lustre: lustre-MDT0000: Denying connection for new client 64e927af-3b15-4fa0-a235-64e7b9e72ef7 (at 192.168.203.22@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 5114.338881] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5114.349207] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5114.349257] Lustre: Skipped 1 previous similar message [ 5114.365898] Lustre: Skipped 1 previous similar message [ 5119.459098] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5119.461975] Lustre: Skipped 1 previous similar message [ 5124.577937] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5124.597154] Lustre: Skipped 1 previous similar message [ 5128.159864] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5128.328822] Lustre: server umount lustre-OST0000 complete [ 5132.421666] LustreError: 168453:0:(ldlm_lib.c:3029:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5132.439951] Lustre: 167801:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5132.450524] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5132.487051] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5132.505511] LustreError: 167801:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000059f3d01c x1875319694867328/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5132.542416] LustreError: 167801:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5132.558165] LustreError: 167801:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5132.570557] Lustre: lustre-MDT0000: Recovery over after 0:28, of 2 clients 0 recovered and 2 were evicted. [ 5133.260696] Lustre: server umount lustre-MDT0000 complete [ 5134.824569] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5134.858869] LustreError: 164714:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5134.887231] LustreError: 164714:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5138.087374] LustreError: 167415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444366 with bad export cookie 3583217781047097664 [ 5138.099459] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5138.103934] LustreError: 167415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5144.534726] Lustre: server umount lustre-MDT0001 complete [ 5157.603357] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 5161.458818] Key type lgssc unregistered [ 5161.816755] LNet: 169205:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5161.834874] LNetError: 169205:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5161.858940] LNet: Removed LNI 192.168.203.122@tcp [ 5163.023509] Key type .llcrypt unregistered [ 5163.025555] Key type ._llcrypt unregistered [ 5187.826034] Key type ._llcrypt registered [ 5187.828463] Key type .llcrypt registered [ 5187.924355] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 5189.028692] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5189.100670] alg: No test for adler32 (adler32-zlib) [ 5190.263041] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 5190.548978] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 5192.223342] Key type lgssc registered [ 5193.903541] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5202.683300] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5202.710339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5204.095106] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5209.149319] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5213.668981] LustreError: 170614:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5213.717206] LustreError: 170614:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5218.429570] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5218.784146] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 5218.981440] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5224.189790] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5232.593800] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5238.777839] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5246.594042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5247.092424] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5249.136841] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5252.465468] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5261.424298] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5268.964410] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5268.989733] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5270.480212] Lustre: server umount lustre-OST0000 complete [ 5275.105077] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5275.114868] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5275.135510] Lustre: Skipped 1 previous similar message [ 5275.148503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5275.151324] Lustre: Skipped 1 previous similar message [ 5279.202563] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5279.203165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5279.217197] Lustre: Skipped 1 previous similar message [ 5281.164790] Lustre: server umount lustre-MDT0000 complete [ 5284.337833] LustreError: 172455:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5284.367699] LustreError: 172455:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5285.463708] LustreError: 170594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444514 with bad export cookie 4454947488190666847 [ 5285.465875] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5285.479151] LustreError: 170594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5285.943318] Lustre: server umount lustre-MDT0001 complete [ 5298.611365] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 5302.397455] Key type lgssc unregistered [ 5302.866764] LNet: 174148:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5302.889156] LNetError: 174148:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5302.915193] LNet: Removed LNI 192.168.203.122@tcp [ 5303.915196] Key type .llcrypt unregistered [ 5303.917898] Key type ._llcrypt unregistered [ 5334.777124] Key type ._llcrypt registered [ 5334.778437] Key type .llcrypt registered [ 5334.882853] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 5336.103446] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5336.126389] alg: No test for adler32 (adler32-zlib) [ 5337.134281] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 5337.450851] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 5339.215220] Key type lgssc registered [ 5340.684623] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5349.035696] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5349.068148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5350.645601] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5356.712705] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5361.131390] LustreError: 175838:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5361.156324] LustreError: 175838:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5365.500452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5365.882453] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5370.531440] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5378.552855] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5384.733597] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5391.828050] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5392.321045] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5396.427520] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5399.189621] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5407.400497] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5421.024761] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5421.035108] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5421.048145] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5421.069355] Lustre: Skipped 1 previous similar message [ 5426.148546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5426.169939] Lustre: Skipped 1 previous similar message [ 5427.076799] Lustre: server umount lustre-OST0000 complete [ 5431.270421] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5431.297156] Lustre: Skipped 1 previous similar message [ 5431.306418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5436.388843] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5436.393804] Lustre: Skipped 2 previous similar messages [ 5437.668174] Lustre: server umount lustre-MDT0000 complete [ 5441.523803] LustreError: 176567:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5441.553075] LustreError: 176567:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5441.580063] LustreError: 178440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444670 with bad export cookie 1781179251524256543 [ 5441.585954] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5441.595447] LustreError: 178440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5442.272479] Lustre: server umount lustre-MDT0001 complete [ 5453.656979] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 5456.482063] Key type lgssc unregistered [ 5456.736655] LNet: 179393:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5456.751926] LNetError: 179393:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5456.768984] LNet: Removed LNI 192.168.203.122@tcp [ 5457.580656] Key type .llcrypt unregistered [ 5457.583724] Key type ._llcrypt unregistered [ 5479.931368] Key type ._llcrypt registered [ 5479.933990] Key type .llcrypt registered [ 5480.034706] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 5481.331868] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5481.387285] alg: No test for adler32 (adler32-zlib) [ 5482.416391] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 5482.645184] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 5484.343153] Key type lgssc registered [ 5485.568348] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5496.656359] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5496.697727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5498.150912] Lustre: *** cfs_fail_loc=135, val=0*** [ 5498.305922] LustreError: 180914:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5498.315633] LustreError: 180914:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.122@tcp: cfg command failed: rc = -2 [ 5498.326946] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5498.326946] [ 5498.358048] LustreError: MGC192.168.203.122@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5498.389177] LustreError: 180875:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5498.405898] LustreError: 180875:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [ 5498.417710] LustreError: 180875:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 5498.426596] LustreError: 180875:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5498.505415] Lustre: server umount lustre-MDT0000 complete [ 5498.511588] LustreError: 180875:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5506.075376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5506.543570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5511.598591] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5516.773158] LustreError: 181286:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5516.787915] LustreError: 181286:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5517.796153] LustreError: 181285:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5522.475028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5522.887789] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5528.184856] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5536.875180] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5542.913547] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5547.362173] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5547.380412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5547.388217] Lustre: Skipped 1 previous similar message [ 5551.586470] Lustre: server umount lustre-MDT0000 complete [ 5552.617587] LustreError: 182264:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5552.649023] LustreError: 182264:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5555.634912] LustreError: 181268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444784 with bad export cookie 4628182497753887440 [ 5555.660858] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5555.973556] Lustre: server umount lustre-MDT0001 complete [ 5567.822792] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 5571.008904] Key type lgssc unregistered [ 5571.430631] LNet: 183605:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5571.453388] LNetError: 183605:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5572.516876] LNet: Removed LNI 192.168.203.122@tcp [ 5573.507337] Key type .llcrypt unregistered [ 5573.514393] Key type ._llcrypt unregistered [ 5596.359642] Key type ._llcrypt registered [ 5596.365645] Key type .llcrypt registered [ 5596.506862] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 5596.859279] Key type .llcrypt unregistered [ 5596.863597] Key type ._llcrypt unregistered [ 5614.155957] Key type ._llcrypt registered [ 5614.157983] Key type .llcrypt registered [ 5614.262463] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 5615.458654] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5615.568442] alg: No test for adler32 (adler32-zlib) [ 5616.812631] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 5617.056857] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 5618.759213] Key type lgssc registered [ 5620.114492] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5628.403662] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5628.425818] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5644.063647] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5649.183344] LustreError: 185731:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.122@tcp: failed processing log, type 4: rc = -110 [ 5675.872646] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5682.614605] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5691.873609] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5698.713540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5699.561321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5699.702250] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5704.688787] LustreError: 185756:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5704.734505] LustreError: 185756:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5706.210641] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5709.795416] LustreError: 185757:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5710.856694] LustreError: 185756:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5710.895335] LustreError: 185756:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5715.918418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5715.943334] LustreError: 185757:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5715.965516] LustreError: 185757:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5716.271115] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5720.762591] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5728.724213] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5734.953508] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5737.986695] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5752.293612] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5752.295988] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5752.306797] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5752.313698] Lustre: Skipped 1 previous similar message [ 5757.424860] LustreError: 185746:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5757.628684] Lustre: server umount lustre-OST0000 complete [ 5767.658549] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5767.662482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5767.684995] Lustre: Skipped 1 previous similar message [ 5767.686831] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5767.690946] Lustre: Skipped 1 previous similar message [ 5772.771068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5772.788078] Lustre: Skipped 2 previous similar messages [ 5777.375429] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5777.654681] Lustre: server umount lustre-MDT0000 complete [ 5777.889507] LustreError: 186773:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5777.905180] LustreError: 186773:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5781.267806] LustreError: 185737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445010 with bad export cookie 16921338929027133439 [ 5781.278870] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5781.286340] LustreError: 185737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5781.517104] Lustre: server umount lustre-MDT0001 complete [ 5792.230973] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 5795.196634] Key type lgssc unregistered [ 5795.630023] LNet: 189743:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5795.647934] LNetError: 189743:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5795.684136] LNet: Removed LNI 192.168.203.122@tcp [ 5796.738242] Key type .llcrypt unregistered [ 5796.740363] Key type ._llcrypt unregistered [ 5827.202480] Key type ._llcrypt registered [ 5827.205313] Key type .llcrypt registered [ 5827.349988] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 5828.125635] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5828.150484] alg: No test for adler32 (adler32-zlib) [ 5829.256280] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 5829.524911] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 5831.207229] Key type lgssc registered [ 5833.330570] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5841.861828] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5841.889698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5843.533769] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5848.679873] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5853.667065] LustreError: 191157:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5853.697916] LustreError: 191157:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5856.861814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5857.249630] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5862.110656] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5869.660512] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5874.667917] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5882.472242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5882.806293] LustreError: 192976:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5882.992446] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5889.516273] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 5890.437651] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5899.005836] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5904.364210] Lustre: Failing over lustre-MDT0000 [ 5904.505499] Lustre: server umount lustre-MDT0000 complete [ 5904.865507] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5904.868525] LustreError: 191884:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5904.871956] Lustre: Skipped 2 previous similar messages [ 5904.883990] LustreError: 191884:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5909.985750] LustreError: 191157:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5910.008650] LustreError: 191157:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5915.107091] LustreError: 191156:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5915.142618] LustreError: 191156:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5923.883713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5924.036152] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5924.499972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5924.570718] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5927.035503] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5929.792696] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5929.967972] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5929.984894] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 5930.029586] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 5935.507389] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 5939.684764] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 5956.067839] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5956.083045] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5956.116228] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5960.160187] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5960.186213] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5960.211409] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5962.149459] Lustre: server umount lustre-OST0000 complete [ 5970.920797] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5970.924718] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5970.933818] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5970.933828] Lustre: Skipped 1 previous similar message [ 5970.947802] Lustre: Skipped 1 previous similar message [ 5972.555760] Lustre: server umount lustre-MDT0000 complete [ 5976.036803] LustreError: 191153:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5976.062420] LustreError: 191153:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 5976.422728] LustreError: 191138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445205 with bad export cookie 16462302708290642928 [ 5976.428190] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5976.435955] LustreError: 191138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5976.919733] Lustre: server umount lustre-MDT0001 complete [ 5988.142210] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 5991.100585] Key type lgssc unregistered [ 5991.393375] LNet: 195976:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5991.401757] LNetError: 195976:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5991.427631] LNet: Removed LNI 192.168.203.122@tcp [ 5992.450292] Key type .llcrypt unregistered [ 5992.461120] Key type ._llcrypt unregistered [ 6016.842617] Key type ._llcrypt registered [ 6016.845831] Key type .llcrypt registered [ 6016.987722] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 6017.945574] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6017.972819] alg: No test for adler32 (adler32-zlib) [ 6019.222212] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 6019.568374] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 6021.255323] Key type lgssc registered [ 6022.295983] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6029.773477] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6029.797828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6031.298244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6035.893365] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6041.582097] LustreError: 197387:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6041.602430] LustreError: 197387:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6043.004894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6043.223091] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6047.643562] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6056.257573] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6063.588883] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6069.957173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6070.297026] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6075.982653] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6076.519561] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6082.379960] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6086.557904] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6093.340466] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6105.270268] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6114.276756] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6114.282748] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6114.295029] Lustre: Skipped 1 previous similar message [ 6114.309845] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6119.402638] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6119.412239] Lustre: Skipped 2 previous similar messages [ 6120.477599] Lustre: server umount lustre-OST0000 complete [ 6124.514168] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6124.517771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6124.522749] Lustre: Skipped 1 previous similar message [ 6124.531187] Lustre: Skipped 1 previous similar message [ 6129.633157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6129.915822] Lustre: server umount lustre-MDT0000 complete [ 6133.670901] LustreError: 197368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445362 with bad export cookie 2380791794612152913 [ 6133.673160] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6133.687604] LustreError: 197368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6134.105769] Lustre: server umount lustre-MDT0001 complete [ 6149.153787] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 6152.236347] Key type lgssc unregistered [ 6152.622735] LNet: 201232:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6152.628642] LNetError: 201232:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6152.644720] LNet: Removed LNI 192.168.203.122@tcp [ 6153.527186] Key type .llcrypt unregistered [ 6153.529063] Key type ._llcrypt unregistered [ 6172.274392] Key type ._llcrypt registered [ 6172.277806] Key type .llcrypt registered [ 6172.443350] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 6173.687949] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6173.729518] alg: No test for adler32 (adler32-zlib) [ 6174.711425] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 6174.925843] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 6176.583643] Key type lgssc registered [ 6177.652404] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6184.707792] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6184.727640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6186.114185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6191.358417] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6195.748759] LustreError: 202495:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6195.763555] LustreError: 202495:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6200.015484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6200.546957] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6204.225428] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6211.030189] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6217.295217] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6224.177589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6224.708599] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6230.915872] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6231.359453] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6238.122965] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6244.054714] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6259.179222] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6264.399945] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6270.466533] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6282.211455] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6282.211620] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6282.236166] LustreError: Skipped 1 previous similar message [ 6282.239691] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6282.251538] Lustre: Skipped 1 previous similar message [ 6282.274428] Lustre: Skipped 1 previous similar message [ 6286.151359] Lustre: server umount lustre-OST0000 complete [ 6292.448685] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6292.462303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6292.468833] Lustre: Skipped 1 previous similar message [ 6296.622302] Lustre: server umount lustre-MDT0000 complete [ 6297.572541] LustreError: 203222:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6297.584528] LustreError: 203222:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6300.153457] LustreError: 202474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445529 with bad export cookie 17790455274698304172 [ 6300.158383] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6300.171390] LustreError: 202474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6300.442253] Lustre: server umount lustre-MDT0001 complete [ 6311.197728] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 6314.131505] Key type lgssc unregistered [ 6314.604594] LNet: 207321:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6314.609507] LNetError: 207321:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6314.624630] LNet: Removed LNI 192.168.203.122@tcp [ 6315.690087] Key type .llcrypt unregistered [ 6315.694061] Key type ._llcrypt unregistered [ 6339.887565] Key type ._llcrypt registered [ 6339.889847] Key type .llcrypt registered [ 6340.016557] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 6340.874955] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6340.972471] alg: No test for adler32 (adler32-zlib) [ 6342.157367] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 6342.671199] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 6344.432770] Key type lgssc registered [ 6345.537819] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6355.660322] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6355.729389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6357.640031] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6364.588038] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6367.212597] LustreError: 208732:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6367.243573] LustreError: 208732:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6372.324290] LustreError: 208731:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6377.448428] LustreError: 208732:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6377.672725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6378.727723] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6383.277841] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6390.504097] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6396.463470] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6403.647713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6404.119367] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6410.199505] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6417.191974] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6427.575540] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6442.987703] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6442.992682] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6443.004176] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6443.011774] Lustre: Skipped 1 previous similar message [ 6446.720643] Lustre: server umount lustre-OST0000 complete [ 6453.218373] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6453.231339] Lustre: Skipped 1 previous similar message [ 6453.235665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6453.242733] Lustre: Skipped 1 previous similar message [ 6457.440674] Lustre: server umount lustre-MDT0000 complete [ 6458.339505] LustreError: 208732:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6458.355819] LustreError: 208732:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6461.447280] LustreError: 208713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445690 with bad export cookie 5835361032775773076 [ 6461.452129] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6461.459990] LustreError: 208713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6461.871715] Lustre: server umount lustre-MDT0001 complete [ 6473.237732] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 6476.260559] Key type lgssc unregistered [ 6476.732836] LNet: 212435:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6476.747576] LNetError: 212435:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6476.822905] LNet: Removed LNI 192.168.203.122@tcp [ 6477.903659] Key type .llcrypt unregistered [ 6477.906750] Key type ._llcrypt unregistered [ 6499.640026] Key type ._llcrypt registered [ 6499.642286] Key type .llcrypt registered [ 6499.767862] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 6500.913234] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6501.159666] alg: No test for adler32 (adler32-zlib) [ 6502.297670] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 6502.504943] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 6504.207251] Key type lgssc registered [ 6505.666583] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6514.047437] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6514.075151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6515.592407] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6520.036519] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6525.920835] LustreError: 213697:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6525.953737] LustreError: 213697:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6528.934858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6529.456876] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6534.543796] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6541.990206] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6547.855597] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6553.685733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6554.182429] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6560.304803] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6560.838740] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6569.015859] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6585.831212] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6585.837072] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6585.837788] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6585.851931] Lustre: Skipped 1 previous similar message [ 6588.661669] Lustre: server umount lustre-OST0000 complete [ 6596.065236] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6596.067409] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6596.076951] LustreError: Skipped 1 previous similar message [ 6596.102232] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6596.113725] Lustre: Skipped 2 previous similar messages [ 6601.184959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6601.196643] Lustre: Skipped 1 previous similar message [ 6605.285955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6605.303560] Lustre: Skipped 1 previous similar message [ 6607.327261] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6607.436899] Lustre: server umount lustre-MDT0000 complete [ 6610.404486] LustreError: 213696:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6610.430849] LustreError: 213696:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6611.193565] LustreError: 213676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445840 with bad export cookie 15476080489169456322 [ 6611.196532] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6611.202622] LustreError: 213676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6611.563911] Lustre: server umount lustre-MDT0001 complete [ 6622.970278] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 6625.437616] Key type lgssc unregistered [ 6625.742720] LNet: 217380:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6625.754122] LNetError: 217380:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6625.781529] LNet: Removed LNI 192.168.203.122@tcp [ 6626.679219] Key type .llcrypt unregistered [ 6626.681297] Key type ._llcrypt unregistered [ 6645.530428] Key type ._llcrypt registered [ 6645.532985] Key type .llcrypt registered [ 6645.630469] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 6646.637859] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6646.672474] alg: No test for adler32 (adler32-zlib) [ 6647.801875] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 6648.041063] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 6649.767803] Key type lgssc registered [ 6651.193536] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6658.389360] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6658.417613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6659.795965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6663.563762] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6669.289851] LustreError: 218645:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6669.306078] LustreError: 218645:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6671.360572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6671.793842] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6675.912189] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6685.187441] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6691.582823] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6699.054339] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6699.459096] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6704.627330] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6706.767048] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6714.881517] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6724.138605] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6724.216633] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6724.437787] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6730.385387] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6737.681625] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6739.183727] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6751.322425] Lustre: Permanently deactivating lustre-OST0001 [ 6751.344797] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6758.474126] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6765.065726] Lustre: Permanently reactivating lustre-OST0001 [ 6765.082550] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6765.087189] Lustre: Skipped 2 previous similar messages [ 6768.178977] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6768.185853] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6768.188629] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6768.199741] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6768.219591] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6781.931818] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6781.940784] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6781.961259] Lustre: Skipped 2 previous similar messages [ 6785.238642] Lustre: server umount lustre-OST0001 complete [ 6787.046702] LustreError: 220470:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6787.081282] LustreError: 220470:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6791.649504] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6791.669508] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6791.707117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6791.725945] Lustre: Skipped 1 previous similar message [ 6792.166949] LustreError: 220471:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6792.199496] LustreError: 220471:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6795.813511] Lustre: server umount lustre-OST0000 complete [ 6799.842089] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6799.860938] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6799.881445] Lustre: Skipped 1 previous similar message [ 6799.892562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6799.896298] Lustre: Skipped 2 previous similar messages [ 6802.400987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6802.419263] Lustre: Skipped 1 previous similar message [ 6805.708772] Lustre: server umount lustre-MDT0000 complete [ 6807.525760] LustreError: 218641:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6807.553062] LustreError: 218641:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6809.501520] LustreError: 218625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446038 with bad export cookie 5662573279991887713 [ 6809.504134] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6809.514521] LustreError: 218625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6809.923642] Lustre: server umount lustre-MDT0001 complete [ 6820.319636] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 6823.535436] Key type lgssc unregistered [ 6823.915746] LNet: 223710:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6823.927204] LNetError: 223710:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6823.944952] LNet: Removed LNI 192.168.203.122@tcp [ 6825.022148] Key type .llcrypt unregistered [ 6825.025320] Key type ._llcrypt unregistered [ 6869.867762] Key type ._llcrypt registered [ 6869.873023] Key type .llcrypt registered [ 6869.963488] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 6871.051940] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6871.128792] alg: No test for adler32 (adler32-zlib) [ 6872.301543] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 6872.627440] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 6874.407175] Key type lgssc registered [ 6875.619863] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6883.776687] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6883.815574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6885.137775] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6885.180834] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6885.316862] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6890.109636] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6898.771566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6898.914895] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6898.960936] Lustre: 225908:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6899.316566] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6903.954210] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6912.160557] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6917.866596] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6923.724322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6923.877026] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6924.091571] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6930.549793] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 6930.875788] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 6939.489873] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6943.354952] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6946.705510] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6953.617472] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6960.461494] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6969.608602] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6988.965914] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6988.977714] Lustre: Skipped 2 previous similar messages [ 7029.834865] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7029.841617] Lustre: Skipped 4 previous similar messages [ 7102.372839] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 7102.378369] Lustre: Skipped 8 previous similar messages [ 7109.600124] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7109.606935] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7109.617103] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7110.121794] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7110.128568] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7110.148138] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7114.716429] Lustre: server umount lustre-OST0000 complete [ 7119.336808] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7119.344799] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7119.357690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7119.363062] Lustre: Skipped 1 previous similar message [ 7125.475551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7125.486221] Lustre: Skipped 3 previous similar messages [ 7130.598104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7130.602518] Lustre: Skipped 1 previous similar message [ 7133.151198] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7133.354344] Lustre: server umount lustre-MDT0000 complete [ 7136.486807] LustreError: 226651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446365 with bad export cookie 12733750991802292426 [ 7136.497220] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7136.501231] LustreError: 226651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7136.760965] Lustre: server umount lustre-MDT0001 complete [ 7147.016396] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 7150.355669] Key type lgssc unregistered [ 7150.704676] LNet: 230724:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7150.724763] LNetError: 230724:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7150.745952] LNet: Removed LNI 192.168.203.122@tcp [ 7151.896947] Key type .llcrypt unregistered [ 7151.899084] Key type ._llcrypt unregistered [ 7177.060230] Key type ._llcrypt registered [ 7177.065420] Key type .llcrypt registered [ 7177.158502] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 7178.046020] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7178.123777] alg: No test for adler32 (adler32-zlib) [ 7179.245678] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 7179.481448] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 7181.231149] Key type lgssc registered [ 7182.368533] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7189.495098] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7189.521509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7190.916630] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7194.618977] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7200.228416] LustreError: 232133:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7200.253623] LustreError: 232133:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7202.930358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7203.204005] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7207.114770] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7214.424325] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7220.171793] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7226.839505] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7227.209873] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7233.128481] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7235.454773] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7239.780872] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7259.108654] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7259.117206] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7259.127406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7259.423678] Lustre: server umount lustre-OST0000 complete [ 7264.234316] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7264.238365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7264.245855] Lustre: Skipped 2 previous similar messages [ 7264.271016] Lustre: Skipped 2 previous similar messages [ 7268.770117] Lustre: server umount lustre-MDT0000 complete [ 7269.345963] LustreError: 232134:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7269.366285] LustreError: 232134:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7270.373054] LustreError: 232133:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7270.400589] LustreError: 232133:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7272.230819] LustreError: 234711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446501 with bad export cookie 5347808266498075427 [ 7272.232283] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7272.250304] LustreError: 234711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7272.578756] Lustre: server umount lustre-MDT0001 complete [ 7282.475070] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 7285.112644] Key type lgssc unregistered [ 7285.375601] LNet: 235824:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7285.382496] LNetError: 235824:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7285.403557] LNet: Removed LNI 192.168.203.122@tcp [ 7286.338672] Key type .llcrypt unregistered [ 7286.341116] Key type ._llcrypt unregistered [ 7307.540232] Key type ._llcrypt registered [ 7307.542180] Key type .llcrypt registered [ 7307.645301] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 7308.524735] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7308.701183] alg: No test for adler32 (adler32-zlib) [ 7309.806414] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 7310.011548] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 7311.639648] Key type lgssc registered [ 7312.694328] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7584.246704] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 7586.489048] Key type lgssc unregistered [ 7586.702511] LNet: 237834:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7586.707261] LNetError: 237834:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7586.720732] LNet: Removed LNI 192.168.203.122@tcp [ 7587.403450] Key type .llcrypt unregistered [ 7587.405745] Key type ._llcrypt unregistered [ 7612.247286] Key type ._llcrypt registered [ 7612.252084] Key type .llcrypt registered [ 7612.371262] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 7629.511758] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 7631.024862] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7631.048693] alg: No test for adler32 (adler32-zlib) [ 7632.133645] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 7632.481509] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 7634.177806] Key type lgssc registered [ 7635.168908] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7664.950756] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7403 sec ========== 10:48:12 (1788446892) [ 7666.821784] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 10:48:13 (1788446893) === [ 7670.803435] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 10:48:17 (1788446897) === [ 7706.421386] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 7708.768710] Key type lgssc unregistered [ 7709.036849] LNet: 243070:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7709.048458] LNetError: 243070:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7709.059487] LNet: Removed LNI 192.168.203.122@tcp [ 7709.925792] Key type .llcrypt unregistered [ 7709.929254] Key type ._llcrypt unregistered