[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 485532893 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.002360] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004012] kvm-guest: setup PV IPIs [ 0.007582] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010131] LSM: Security Framework initializing [ 0.011062] Yama: becoming mindful. [ 0.012004] SELinux: Initializing. [ 0.012848] *** VALIDATE selinux *** [ 0.019822] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.023780] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.024160] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025124] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027093] *** VALIDATE tmpfs *** [ 0.029024] *** VALIDATE proc *** [ 0.030250] *** VALIDATE cgroup *** [ 0.031010] *** VALIDATE cgroup2 *** [ 0.032271] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034090] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036029] Spectre V2 : User space: Vulnerable [ 0.037010] Speculative Store Bypass: Vulnerable [ 0.040488] debug: unmapping init [mem 0xffffffffae459000-0xffffffffae460fff] [ 0.042182] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043778] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044023] ... version: 2 [ 0.045011] ... bit width: 48 [ 0.046012] ... generic registers: 4 [ 0.047014] ... value mask: 0000ffffffffffff [ 0.048014] ... max period: 00007fffffffffff [ 0.049014] ... fixed-purpose events: 3 [ 0.050007] ... event mask: 000000070000000f [ 0.051294] rcu: Hierarchical SRCU implementation. [ 0.053486] smp: Bringing up secondary CPUs ... [ 0.054586] x86: Booting SMP configuration: [ 0.055017] .... node #0, CPUs: #1 #2 #3 [ 0.060076] smp: Brought up 1 node, 4 CPUs [ 0.062025] smpboot: Max logical packages: 1 [ 0.063013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.155057] node 0 deferred pages initialised in 90ms [ 0.162178] devtmpfs: initialized [ 0.164223] x86/mm: Memory block size: 128MB [ 0.165000] gcov: version magic: 0x41383552 [ 0.171295] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.172072] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.173253] pinctrl core: initialized pinctrl subsystem [ 0.174236] [ 0.174866] ************************************************************* [ 0.177020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.180018] ** ** [ 0.183013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.186016] ** ** [ 0.189015] ** This means that this kernel is built to expose internal ** [ 0.192015] ** IOMMU data structures, which may compromise security on ** [ 0.195068] ** your system. ** [ 0.197015] ** ** [ 0.200018] ** If you see this message and you are not debugging the ** [ 0.203019] ** kernel, report this immediately to your vendor! ** [ 0.206029] ** ** [ 0.209023] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.211032] ************************************************************* [ 0.215188] NET: Registered protocol family 16 [ 0.219758] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.223064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.226074] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.230062] cpuidle: using governor menu [ 0.232381] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.236232] PCI: Using configuration type 1 for base access [ 0.238136] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.246379] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.248017] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.252016] cryptd: max_cpu_qlen set to 1000 [ 0.253292] ACPI: Added _OSI(Module Device) [ 0.255055] ACPI: Added _OSI(Processor Device) [ 0.256011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.258012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.261693] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.268686] ACPI: Interpreter enabled [ 0.270066] ACPI: PM: (supports S0 S3 S4 S5) [ 0.273012] ACPI: Using IOAPIC for interrupt routing [ 0.276950] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.284831] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.304212] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.307092] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.312021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.317226] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.326371] acpiphp: Slot [2] registered [ 0.328359] acpiphp: Slot [5] registered [ 0.331278] acpiphp: Slot [6] registered [ 0.333310] acpiphp: Slot [7] registered [ 0.335377] acpiphp: Slot [8] registered [ 0.338129] acpiphp: Slot [9] registered [ 0.340094] acpiphp: Slot [10] registered [ 0.342101] acpiphp: Slot [3] registered [ 0.344170] acpiphp: Slot [4] registered [ 0.345108] acpiphp: Slot [11] registered [ 0.347099] acpiphp: Slot [12] registered [ 0.349151] acpiphp: Slot [13] registered [ 0.350106] acpiphp: Slot [14] registered [ 0.352105] acpiphp: Slot [15] registered [ 0.354136] acpiphp: Slot [16] registered [ 0.355101] acpiphp: Slot [17] registered [ 0.357219] acpiphp: Slot [18] registered [ 0.359173] acpiphp: Slot [19] registered [ 0.361122] acpiphp: Slot [20] registered [ 0.363162] acpiphp: Slot [21] registered [ 0.364081] acpiphp: Slot [22] registered [ 0.366103] acpiphp: Slot [23] registered [ 0.367096] acpiphp: Slot [24] registered [ 0.369092] acpiphp: Slot [25] registered [ 0.371087] acpiphp: Slot [26] registered [ 0.372090] acpiphp: Slot [27] registered [ 0.374176] acpiphp: Slot [28] registered [ 0.376126] acpiphp: Slot [29] registered [ 0.377091] acpiphp: Slot [30] registered [ 0.379095] acpiphp: Slot [31] registered [ 0.380080] PCI host bridge to bus 0000:00 [ 0.382021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.385023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.388027] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.390063] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.393076] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.396022] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.398174] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.402088] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.481355] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.492868] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.499053] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.502019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.506021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.510019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.513740] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.516908] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.520039] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.522811] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.528015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.539993] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.545017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.552456] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.563018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.575018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.603018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.622631] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.648145] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.661017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.700068] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.720194] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.749022] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.765020] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.806020] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.826826] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.864019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.872019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.897020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.915089] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.925022] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.936018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.960018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.975274] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.987016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.999020] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 1.026017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 1.039072] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 1.042354] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 1.044348] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 1.047410] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 1.049206] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 1.053120] iommu: Default domain type: Passthrough [ 1.056559] SCSI subsystem initialized [ 1.058125] ACPI: bus type USB registered [ 1.059113] usbcore: registered new interface driver usbfs [ 1.061053] usbcore: registered new interface driver hub [ 1.062073] usbcore: registered new device driver usb [ 1.063212] pps_core: LinuxPPS API ver. 1 registered [ 1.064007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.066049] PTP clock support registered [ 1.068094] EDAC MC: Ver: 3.0.0 [ 1.069585] PCI: Using ACPI for IRQ routing [ 1.071458] NetLabel: Initializing [ 1.072011] NetLabel: domain hash size = 128 [ 1.074007] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.075084] NetLabel: unlabeled traffic allowed by default [ 1.076318] vgaarb: loaded [ 1.078318] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.079010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.085541] clocksource: Switched to clocksource kvm-clock [ 1.204598] VFS: Disk quotas dquot_6.6.0 [ 1.206079] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.214742] *** VALIDATE ramfs *** [ 1.215818] *** VALIDATE hugetlbfs *** [ 1.218372] pnp: PnP ACPI init [ 1.220766] pnp: PnP ACPI: found 6 devices [ 1.236448] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.240498] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.242763] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.245751] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.248676] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.251802] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.255750] NET: Registered protocol family 2 [ 1.258820] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.264751] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.268628] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.272945] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.275981] TCP: Hash tables configured (established 65536 bind 65536) [ 1.278041] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.280114] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.282034] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.284269] NET: Registered protocol family 1 [ 1.286753] RPC: Registered named UNIX socket transport module. [ 1.288718] RPC: Registered udp transport module. [ 1.290157] RPC: Registered tcp transport module. [ 1.291733] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.293915] NET: Registered protocol family 44 [ 1.295505] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.297677] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.299432] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.301533] PCI: CLS 0 bytes, default 64 [ 1.303399] Unpacking initramfs... [ 2.907842] debug: unmapping init [mem 0xffff99f4fcc54000-0xffff99f4fffbffff] [ 2.911685] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.914044] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.917284] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.567215] Initialise system trusted keyrings [ 3.569260] Key type blacklist registered [ 3.575050] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.590808] zbud: loaded [ 3.594294] *** VALIDATE nfs *** [ 3.595678] *** VALIDATE nfs4 *** [ 3.597363] pstore: using deflate compression [ 3.600360] Platform Keyring initialized [ 3.715464] NET: Registered protocol family 38 [ 3.716916] Key type asymmetric registered [ 3.717955] Asymmetric key parser 'x509' registered [ 3.721796] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.725345] io scheduler mq-deadline registered [ 3.727060] io scheduler kyber registered [ 3.729240] io scheduler bfq registered [ 3.731157] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.734246] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.737566] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.740384] ACPI: Power Button [PWRF] [ 3.746797] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.754296] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.769969] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.778627] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.795962] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.821987] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.850214] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.854992] Non-volatile memory driver v1.3 [ 3.856503] Linux agpgart interface v0.103 [ 3.888916] virtio_blk virtio1: [vda] 149960 512-byte logical blocks (76.8 MB/73.2 MiB) [ 3.891760] vda: detected capacity change from 0 to 76779520 [ 3.905975] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.908790] vdb: detected capacity change from 0 to 1073741824 [ 3.924582] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.927607] vdc: detected capacity change from 0 to 2621440000 [ 3.943529] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.948333] vdd: detected capacity change from 0 to 2621440000 [ 3.964073] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.966991] vde: detected capacity change from 0 to 4294967296 [ 3.986312] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.989142] vdf: detected capacity change from 0 to 4294967296 [ 4.003830] libphy: Fixed MDIO Bus: probed [ 4.010584] usbcore: registered new interface driver usbserial_generic [ 4.013160] usbserial: USB Serial support registered for generic [ 4.016785] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.024662] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.026559] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.028645] mousedev: PS/2 mouse device common for all mice [ 4.031771] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.033816] rtc_cmos 00:05: RTC can wake from S4 [ 4.039784] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.045391] rtc_cmos 00:05: registered as rtc0 [ 4.048904] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.051420] intel_pstate: CPU model not supported [ 4.054545] hid: raw HID events driver (C) Jiri Kosina [ 4.056095] usbcore: registered new interface driver usbhid [ 4.057722] usbhid: USB HID core driver [ 4.058911] drop_monitor: Initializing network drop monitor service [ 4.058977] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.061238] Initializing XFRM netlink socket [ 4.065786] NET: Registered protocol family 10 [ 4.068412] Segment Routing with IPv6 [ 4.070023] NET: Registered protocol family 17 [ 4.072033] mpls_gso: MPLS GSO support [ 4.078402] RAS: Correctable Errors collector initialized. [ 4.081444] AVX version of gcm_enc/dec engaged. [ 4.083987] AES CTR mode by8 optimization enabled [ 4.169755] sched_clock: Marking stable (4169735941, 0)->(5188568178, -1018832237) [ 4.172610] registered taskstats version 1 [ 4.174808] Loading compiled-in X.509 certificates [ 4.177506] zswap: loaded using pool lzo/zbud [ 4.208773] Key type big_key registered [ 4.222763] Key type encrypted registered [ 4.223786] ima: No TPM chip found, activating TPM-bypass! [ 4.225018] ima: Allocated hash algorithm: sha1 [ 4.226053] ima: No architecture policies found [ 4.227519] evm: Initialising EVM extended attributes: [ 4.229095] evm: security.selinux [ 4.230126] evm: security.ima [ 4.230969] evm: security.capability [ 4.232241] evm: HMAC attrs: 0x1 [ 4.234488] rtc_cmos 00:05: setting system clock to 2026-09-08 06:57:44 UTC (1788850664) [ 4.238652] debug: unmapping init [mem 0xffffffffaf403000-0xffffffffaf5fffff] [ 4.240911] debug: unmapping init [mem 0xffffffffae182000-0xffffffffae458fff] [ 4.248148] Write protecting the kernel read-only data: 28672k [ 4.252228] debug: unmapping init [mem 0xffffffffac803000-0xffffffffac9fffff] [ 4.254936] debug: unmapping init [mem 0xffffffffad114000-0xffffffffad1fffff] [ 4.291778] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.301570] systemd[1]: Detected virtualization kvm. [ 4.302875] systemd[1]: Detected architecture x86-64. [ 4.304092] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.330388] systemd[1]: No hostname configured. [ 4.332328] systemd[1]: Set hostname to . [ 4.334481] random: systemd: uninitialized urandom read (16 bytes read) [ 4.337285] systemd[1]: Initializing machine ID from random generator. [ 4.508381] random: ln: uninitialized urandom read (6 bytes read) [ 4.612354] random: systemd: uninitialized urandom read (16 bytes read) [ 4.614960] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 4.620186] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 4.626438] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Initrd Root Device. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.491538] device-mapper: uevent: version 1.0.3 [ 5.493904] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 6.405738] virtio_net virtio0 ens2: renamed from eth0 [ 6.425452] random: fast init done [ 6.487461] scsi host0: ata_piix [ 6.500579] scsi host1: ata_piix [ 6.501697] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 6.504068] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 11.738056] random: crng init done [ 11.739847] random: 7 urandom warning(s) missed due to ratelimiting [ 14.222188] dracut-initqueue[585]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 16.335554] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 20.667958] printk: systemd: 26 output lines suppressed due to ratelimiting [ 22.109183] SELinux: Disabled at runtime. [ 22.320711] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 22.346867] systemd[1]: Detected virtualization kvm. [ 22.354366] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 24.480814] systemd[1]: initrd-switch-root.service: Succeeded. [ 24.507041] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 24.534471] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 24.543995] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 24.552340] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 24.574553] systemd[1]: Starting Journal Service... Starting Journal Service... [ 24.589513] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... [ 24.665113] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Kernel Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Switch Root. Starting Create list of required st…ce nodes for the current kernel... Starting Remount Root and Kernel File Systems... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ OK ] Listening on RPCbind Server Activation Socket. Starting udev Coldplug all Devices... [ OK ] Created slice User and Session Slice. [ OK ] Stopped target Initrd File Systems. Mounting Kernel Debug File System... [ OK ] Stopped target Initrd Root File System. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. Mounting Huge Pages File System... [ OK ] Reached target Local Encrypted Volumes. Mounting POSIX Message Queue File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 26.506547] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 27.660850] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 27.806545] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 28.569371] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 28.739592] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ ***] A start job is running for Configur…only root support (11s / no limit)[ 35.890714] Key type dns_resolver registered [ **] A start job is running for Configur…only root support (11s / no limit) [ *] A start job is running for Configur…only root support (12s / no limit) [ **] A start job is running for Configur…only root support (12s / no limit)[ 37.239628] NFS: Registering the id_resolver key type [ 37.241692] Key type id_resolver registered [ 37.247021] Key type id_legacy registered [ ***] A start job is running for Configur…only root support (13s / no limit) [ *** ] A start job is running for Configur…only root support (13s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Login Service. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg233-server login: [ 56.256032] hrtimer: interrupt took 7562169 ns [ 102.655522] libcfs: loading out-of-tree module taints kernel. [ 102.699682] Key type ._llcrypt registered [ 102.709388] Key type .llcrypt registered [ 102.818311] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 120.727452] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 122.322515] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 122.332664] alg: No test for adler32 (adler32-zlib) [ 123.699777] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 124.395638] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 126.112633] Key type lgssc registered [ 127.962417] Lustre: Echo OBD driver; http://www.lustre.org/ [ 140.967518] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 180.885591] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 194.307350] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 194.341816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 195.650687] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 195.692781] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 195.802226] Lustre: lustre-MDT0000: new disk, initializing [ 195.941411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 195.976660] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 200.426330] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 213.866118] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 213.985467] Lustre: 6514:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 214.002216] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 214.006671] Lustre: Skipped 1 previous similar message [ 214.096769] Lustre: lustre-MDT0001: new disk, initializing [ 214.179680] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 214.221477] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 214.249161] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 218.959360] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 223.701771] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 233.829996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 234.084745] Lustre: lustre-OST0000: new disk, initializing [ 234.088154] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 234.097258] Lustre: 8467:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 234.224417] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 239.606739] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 239.625876] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 239.772165] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 243.111810] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 258.151859] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 258.343704] Lustre: lustre-OST0001: new disk, initializing [ 258.353607] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 258.371253] Lustre: 9539:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 258.482777] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 264.740667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 264.754914] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 264.818278] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 265.629195] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 279.424751] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 290.826935] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 297.710931] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing check_logdir /tmp/testlogs/ [ 303.501913] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing yml_node [ 307.275018] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 309.700611] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 312.012866] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 313.530664] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Sep 8 03:02:52 EDT 2026 [ 330.947484] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 332.590787] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 351.718909] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 351.722635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 351.730489] Lustre: Skipped 3 previous similar messages [ 351.739293] Lustre: Skipped 3 previous similar messages [ 354.558553] Lustre: server umount lustre-MDT0000 complete [ 358.993365] LustreError: 6506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851019 with bad export cookie 9454251918408756977 [ 358.999953] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 359.004113] LustreError: 6506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 361.958633] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 361.958951] LustreError: 8476:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 361.964484] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 361.974386] Lustre: Skipped 1 previous similar message [ 362.009330] LustreError: 8476:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 364.001211] LustreError: 11298:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 364.004350] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 364.059838] Lustre: Skipped 2 previous similar messages [ 364.063911] LustreError: 11298:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 365.458300] Lustre: server umount lustre-MDT0001 complete [ 369.851965] Lustre: server umount lustre-OST0000 complete [ 373.712734] Lustre: server umount lustre-OST0001 complete [ 380.552554] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 390.797418] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 428.434649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 428.828569] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 428.857570] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 428.923632] Lustre: lustre-MDT0000: new disk, initializing [ 429.001238] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 429.020412] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 434.123158] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 444.337716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 444.553702] Lustre: 15095:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 444.711057] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 444.720477] Lustre: Skipped 1 previous similar message [ 444.836309] Lustre: lustre-MDT0001: new disk, initializing [ 444.953277] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 445.016888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 445.031276] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 449.818361] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 459.346740] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 464.627208] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 471.173419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 471.543349] Lustre: lustre-OST0000: new disk, initializing [ 471.550570] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 471.573333] Lustre: 17061:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 471.727506] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 472.864753] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 472.871735] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 473.003077] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 478.378415] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 488.428599] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 493.429419] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 493.672157] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 498.052540] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 498.177880] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 503.778851] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 503.780660] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 503.790111] LustreError: Skipped 1 previous similar message [ 503.803780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 508.904468] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 508.927704] Lustre: Skipped 2 previous similar messages [ 514.017514] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 514.312420] Lustre: server umount lustre-OST0000 complete [ 519.138832] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 519.141546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 519.152977] Lustre: Skipped 2 previous similar messages [ 519.174951] Lustre: Skipped 3 previous similar messages [ 525.267580] Lustre: server umount lustre-MDT0000 complete [ 529.308565] LustreError: 15089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851189 with bad export cookie 9454251918408758510 [ 529.321959] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 529.590200] Lustre: server umount lustre-MDT0001 complete [ 533.001815] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 03:06:31 (1788851191) [ 540.738762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 541.374916] LustreError: 19326:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 541.495792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 546.097443] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 546.785635] LustreError: 19327:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 550.891286] LustreError: 19326:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 554.322922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 558.480238] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 565.292906] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 571.514728] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 578.264407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 585.900708] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 593.965530] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 601.571868] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 601.575207] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 601.582171] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 601.592134] Lustre: Skipped 1 previous similar message [ 601.596502] Lustre: Skipped 2 previous similar messages [ 606.693049] LustreError: 21147:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 606.889951] Lustre: server umount lustre-OST0000 complete [ 611.810835] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 616.418839] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 618.289249] Lustre: server umount lustre-MDT0000 complete [ 622.404506] LustreError: 19308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851282 with bad export cookie 9454251918408759392 [ 622.414673] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 622.422808] LustreError: 19308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 622.946837] Lustre: server umount lustre-MDT0001 complete [ 636.433970] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 640.064665] Key type lgssc unregistered [ 640.439430] LNet: 22858:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 640.451884] LNetError: 22858:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 640.467252] LNet: Removed LNI 192.168.202.133@tcp [ 641.730934] Key type .llcrypt unregistered [ 641.736980] Key type ._llcrypt unregistered [ 667.640648] Key type ._llcrypt registered [ 667.641825] Key type .llcrypt registered [ 667.729867] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 668.711355] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 668.858930] alg: No test for adler32 (adler32-zlib) [ 669.953318] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 670.188420] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 671.824246] Key type lgssc registered [ 672.900390] Lustre: Echo OBD driver; http://www.lustre.org/ [ 680.145399] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 680.158168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 681.568512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 685.604356] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 691.181018] LustreError: 24268:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 691.197327] LustreError: 24268:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 692.939390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 693.358801] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 697.922848] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 705.770667] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 711.251926] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 716.835872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 717.227527] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 725.085776] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 726.463628] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 732.918103] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 749.025502] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 749.037595] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 749.066476] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 750.765170] Lustre: server umount lustre-OST0000 complete [ 757.222594] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 757.242115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 762.338591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 762.351727] Lustre: Skipped 2 previous similar messages [ 765.510760] Lustre: server umount lustre-MDT0000 complete [ 767.459080] LustreError: 24268:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 767.478951] LustreError: 24268:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 768.565241] LustreError: 24248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851428 with bad export cookie 1985893641987436590 [ 768.571115] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 768.587374] LustreError: 24248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 768.989712] Lustre: server umount lustre-MDT0001 complete [ 781.103658] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 785.334989] Key type lgssc unregistered [ 785.560811] LNet: 28242:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 785.565720] LNetError: 28242:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 785.578670] LNet: Removed LNI 192.168.202.133@tcp [ 786.373777] Key type .llcrypt unregistered [ 786.378580] Key type ._llcrypt unregistered [ 811.348806] Key type ._llcrypt registered [ 811.350774] Key type .llcrypt registered [ 811.433515] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 812.638755] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 812.683795] alg: No test for adler32 (adler32-zlib) [ 813.805489] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 813.985202] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 815.624400] Key type lgssc registered [ 817.075904] Lustre: Echo OBD driver; http://www.lustre.org/ [ 826.042829] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 826.104813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 827.844929] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 833.164292] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 838.115258] LustreError: 29652:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 838.133252] LustreError: 29652:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 842.962669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 843.330529] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 848.824690] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 857.101974] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 863.741619] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 878.452147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 878.816979] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 886.653407] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 895.325372] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 898.278502] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 908.785891] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 908.806267] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 908.838810] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 908.850576] Lustre: Skipped 1 previous similar message [ 908.916171] Lustre: Skipped 1 previous similar message [ 911.441674] Lustre: server umount lustre-OST0000 complete [ 919.012666] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 919.032457] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 919.038086] Lustre: Skipped 1 previous similar message [ 919.055253] Lustre: Skipped 1 previous similar message [ 923.160728] Lustre: server umount lustre-MDT0000 complete [ 924.129702] LustreError: 29648:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 924.149948] LustreError: 29648:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 927.056895] LustreError: 29634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851587 with bad export cookie 18350123541980303186 [ 927.059696] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 927.071948] LustreError: 29634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 927.439915] Lustre: server umount lustre-MDT0001 complete [ 938.321722] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 941.061415] Key type lgssc unregistered [ 941.533701] LNet: 33534:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 941.556560] LNetError: 33534:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 942.573630] LNet: Removed LNI 192.168.202.133@tcp [ 944.024838] Key type .llcrypt unregistered [ 944.028829] Key type ._llcrypt unregistered [ 973.407074] Key type ._llcrypt registered [ 973.409376] Key type .llcrypt registered [ 973.521559] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 974.701119] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 974.771799] alg: No test for adler32 (adler32-zlib) [ 976.220838] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 976.685317] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 978.520290] Key type lgssc registered [ 980.988637] Lustre: Echo OBD driver; http://www.lustre.org/ [ 990.016523] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 990.044875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 991.636673] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 997.244930] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1000.937411] LustreError: 34945:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1000.949864] LustreError: 34945:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1006.058448] LustreError: 34946:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1006.596778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1007.494262] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1013.811319] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1023.797636] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1029.994549] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1037.395948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1037.616616] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1046.352881] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1055.015839] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1061.182498] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1071.588538] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1071.599249] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1071.604657] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1071.614979] Lustre: Skipped 1 previous similar message [ 1073.718936] Lustre: server umount lustre-OST0000 complete [ 1079.273339] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1079.287239] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1079.323485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1079.326120] Lustre: Skipped 1 previous similar message [ 1081.825243] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1081.826951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1081.853793] Lustre: Skipped 1 previous similar message [ 1083.841568] Lustre: server umount lustre-MDT0000 complete [ 1086.955517] LustreError: 34940:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1087.002689] LustreError: 34940:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1088.551323] LustreError: 34925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851748 with bad export cookie 12125608577123623088 [ 1088.557332] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1088.577632] LustreError: 34925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1089.212877] Lustre: server umount lustre-MDT0001 complete [ 1101.805970] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 1105.057354] Key type lgssc unregistered [ 1105.459268] LNet: 38511:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1105.465119] LNetError: 38511:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1105.489178] LNet: Removed LNI 192.168.202.133@tcp [ 1106.690789] Key type .llcrypt unregistered [ 1106.693370] Key type ._llcrypt unregistered [ 1133.121251] Key type ._llcrypt registered [ 1133.125148] Key type .llcrypt registered [ 1133.332379] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 1134.649852] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1134.767701] alg: No test for adler32 (adler32-zlib) [ 1135.792176] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 1136.101441] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 1137.832461] Key type lgssc registered [ 1139.351105] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1146.992274] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1147.017337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1148.540480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1153.484469] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1158.119971] LustreError: 39922:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1158.133298] LustreError: 39922:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1162.525731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1162.966765] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1167.622603] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1175.620269] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1181.185374] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1190.358358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1191.018561] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1193.096346] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1198.827534] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1207.869252] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1215.849875] Lustre: lustre-OST0000: Not available for connect from 192.168.202.33@tcp (stopping) [ 1219.045123] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1219.074101] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1219.085528] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1219.110435] Lustre: Skipped 1 previous similar message [ 1220.958323] Lustre: lustre-OST0000: Not available for connect from 192.168.202.33@tcp (stopping) [ 1221.909144] Lustre: server umount lustre-OST0000 complete [ 1244.642316] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1244.643699] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1244.650796] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1244.655389] Lustre: Skipped 2 previous similar messages [ 1247.673592] Lustre: server umount lustre-MDT0000 complete [ 1249.764945] LustreError: 39917:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1249.791636] LustreError: 39917:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1252.631917] LustreError: 39903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851912 with bad export cookie 10735593394395757569 [ 1252.640363] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1252.650940] LustreError: 39903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1259.269745] Lustre: server umount lustre-MDT0001 complete [ 1271.077872] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 1273.809711] Key type lgssc unregistered [ 1274.070977] LNet: 43629:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1274.076149] LNetError: 43629:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1274.090818] LNet: Removed LNI 192.168.202.133@tcp [ 1274.740120] Key type .llcrypt unregistered [ 1274.742463] Key type ._llcrypt unregistered [ 1298.575346] Key type ._llcrypt registered [ 1298.581100] Key type .llcrypt registered [ 1298.716582] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 1300.637620] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1300.708455] alg: No test for adler32 (adler32-zlib) [ 1301.883698] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 1302.182722] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 1304.008984] Key type lgssc registered [ 1305.620798] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1316.433163] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1316.467928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1318.228586] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1324.755284] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1328.618869] LustreError: 45040:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1328.641112] LustreError: 45040:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1333.751310] LustreError: 45041:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1334.644383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1335.187483] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1340.212665] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1348.201313] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1354.203352] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1360.928918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1361.371980] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1363.444479] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1367.508832] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1376.728389] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1386.482664] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1386.492300] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1386.504376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1388.910748] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.33@tcp (stopping) [ 1390.051817] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1390.051945] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1390.062763] Lustre: Skipped 1 previous similar message [ 1390.074234] Lustre: Skipped 2 previous similar messages [ 1392.073704] Lustre: server umount lustre-MDT0000 complete [ 1393.991234] LustreError: 45037:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1395.383692] LustreError: 47620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852055 with bad export cookie 6444599199400548908 [ 1395.401763] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1395.406147] LustreError: 47620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1399.117098] LustreError: 45035:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1399.142557] LustreError: 45035:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1399.146477] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.33@tcp (stopping) [ 1400.291507] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1401.849147] Lustre: server umount lustre-MDT0001 complete [ 1416.833635] Lustre: server umount lustre-OST0000 complete [ 1438.676234] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 1443.746462] Key type lgssc unregistered [ 1444.468201] LNet: 48903:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1444.480743] LNetError: 48903:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1444.528620] LNet: Removed LNI 192.168.202.133@tcp [ 1446.190874] Key type .llcrypt unregistered [ 1446.192438] Key type ._llcrypt unregistered [ 1475.320832] Key type ._llcrypt registered [ 1475.322113] Key type .llcrypt registered [ 1475.411316] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 1476.542253] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1476.561348] alg: No test for adler32 (adler32-zlib) [ 1477.801791] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 1478.028990] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 1479.680267] Key type lgssc registered [ 1480.818940] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1491.517022] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1491.580685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1507.425078] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1512.544411] LustreError: 50286:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.133@tcp: failed processing log, type 4: rc = -110 [ 1539.232393] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1545.637852] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1554.063489] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1718.614873] Lustre: server umount lustre-OST0000 complete [ 1739.225148] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 1742.538419] Key type lgssc unregistered [ 1742.942757] LNet: 51963:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1742.950516] LNetError: 51963:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1742.978973] LNet: Removed LNI 192.168.202.133@tcp [ 1744.220518] Key type .llcrypt unregistered [ 1744.226425] Key type ._llcrypt unregistered [ 1771.938873] Key type ._llcrypt registered [ 1771.941765] Key type .llcrypt registered [ 1772.089553] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 1773.719091] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1773.769895] alg: No test for adler32 (adler32-zlib) [ 1774.843232] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 1775.201437] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 1776.914772] Key type lgssc registered [ 1778.115712] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1786.442215] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1786.455579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1788.123801] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1793.928474] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1798.634533] LustreError: 53372:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1798.656758] LustreError: 53372:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1803.662036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1803.750760] LustreError: 53373:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1804.150750] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1808.803693] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1818.680342] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1828.590928] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1837.784434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1838.424587] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1845.446756] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1852.659691] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1858.546190] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1858.558747] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1861.735865] Lustre: server umount lustre-OST0000 complete [ 1868.777289] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1868.803647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1868.806505] Lustre: Skipped 1 previous similar message [ 1872.489779] Lustre: server umount lustre-MDT0000 complete [ 1873.898547] LustreError: 53372:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1873.916029] LustreError: 53372:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1876.054259] LustreError: 53355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852536 with bad export cookie 4639892317389146200 [ 1876.063160] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1876.068549] LustreError: 53355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1876.444801] Lustre: server umount lustre-MDT0001 complete [ 1892.048523] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 1895.782980] Key type lgssc unregistered [ 1896.181813] LNet: 56915:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1896.187683] LNetError: 56915:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1896.208547] LNet: Removed LNI 192.168.202.133@tcp [ 1897.234085] Key type .llcrypt unregistered [ 1897.241441] Key type ._llcrypt unregistered [ 1921.408291] Key type ._llcrypt registered [ 1921.410786] Key type .llcrypt registered [ 1921.634310] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 1923.558908] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1923.640688] alg: No test for adler32 (adler32-zlib) [ 1925.094427] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 1925.352220] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 1927.056580] Key type lgssc registered [ 1928.793409] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1937.680242] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1937.705715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1953.312493] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1958.434164] LustreError: 58297:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.133@tcp: failed processing log, type 4: rc = -110 [ 1985.121095] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1991.558764] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1999.837304] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2007.643403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2008.473290] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2008.572682] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 2013.383117] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2013.674373] LustreError: 58322:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2013.707890] LustreError: 58322:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2018.786049] LustreError: 58323:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2019.815430] LustreError: 59335:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2019.831822] LustreError: 59335:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2022.715958] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2023.001075] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2027.320236] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2035.030056] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2038.940960] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2043.363852] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2043.374419] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2043.380747] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2048.497926] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2048.527759] Lustre: Skipped 2 previous similar messages [ 2053.602447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2053.613557] Lustre: Skipped 1 previous similar message [ 2055.136142] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2055.283834] Lustre: server umount lustre-OST0000 complete [ 2079.201935] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2079.204218] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2079.205614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2079.229475] Lustre: Skipped 2 previous similar messages [ 2083.364025] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2083.378232] Lustre: Skipped 2 previous similar messages [ 2085.288236] Lustre: server umount lustre-MDT0000 complete [ 2088.422524] LustreError: 59339:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2088.442868] LustreError: 59339:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2090.133360] LustreError: 58303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852750 with bad export cookie 18419639897122241576 [ 2090.139841] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2090.158347] LustreError: 58303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2096.592691] Lustre: server umount lustre-MDT0001 complete [ 2108.297748] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 2112.668317] Key type lgssc unregistered [ 2113.227694] LNet: 62045:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2113.235461] LNetError: 62045:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2113.266792] LNet: Removed LNI 192.168.202.133@tcp [ 2114.649044] Key type .llcrypt unregistered [ 2114.650839] Key type ._llcrypt unregistered [ 2146.134610] Key type ._llcrypt registered [ 2146.136850] Key type .llcrypt registered [ 2146.209611] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 2147.696249] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2147.796607] alg: No test for adler32 (adler32-zlib) [ 2148.972777] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 2149.305728] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 2150.984465] Key type lgssc registered [ 2152.455791] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2160.025768] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2160.041596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2161.522761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2166.706835] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2170.851787] LustreError: 63454:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2170.886379] LustreError: 63454:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2175.978075] LustreError: 63455:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2178.420369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2179.161208] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2186.502229] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2195.432539] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2201.978378] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2209.626506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2210.179780] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2217.438840] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2226.360453] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2231.668742] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2245.090166] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2245.120693] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2245.169679] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2247.162027] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2247.177428] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2247.206257] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2249.014338] Lustre: server umount lustre-OST0000 complete [ 2255.841850] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2255.844937] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2255.869831] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2255.889927] Lustre: Skipped 1 previous similar message [ 2258.965878] Lustre: server umount lustre-MDT0000 complete [ 2262.498492] LustreError: 64544:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2262.527244] LustreError: 64544:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2263.074754] LustreError: 63434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852923 with bad export cookie 9754314593591371045 [ 2263.076025] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2263.085465] LustreError: 63434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2263.413101] Lustre: server umount lustre-MDT0001 complete [ 2276.704265] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 2281.376633] Key type lgssc unregistered [ 2281.843730] LNet: 67014:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2281.861409] LNetError: 67014:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2281.880989] LNet: Removed LNI 192.168.202.133@tcp [ 2283.167572] Key type .llcrypt unregistered [ 2283.173773] Key type ._llcrypt unregistered [ 2311.218147] Key type ._llcrypt registered [ 2311.220638] Key type .llcrypt registered [ 2311.356146] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 2313.568278] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2313.680019] alg: No test for adler32 (adler32-zlib) [ 2314.842825] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 2315.108136] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 2316.800188] Key type lgssc registered [ 2318.131485] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2328.543417] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 03:36:27 (1788852987) [ 2337.262182] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2337.299067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2338.974250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2345.224325] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2349.543687] LustreError: 68706:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2349.559398] LustreError: 68706:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2354.668283] LustreError: 68705:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2355.312344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2355.782721] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2361.150511] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2368.873445] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2374.963642] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2382.268240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2382.565517] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2384.626315] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2388.500341] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2398.331910] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2405.685301] Lustre: Failing over lustre-MDT0000 [ 2405.914662] Lustre: server umount lustre-MDT0000 complete [ 2406.233112] LustreError: 68700:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2406.888398] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2406.895024] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2410.476245] LustreError: 68705:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2410.477761] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2410.503634] LustreError: 68705:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2415.543758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2415.590101] LustreError: 68705:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2415.622632] LustreError: 68705:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 2415.827749] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2416.124759] Lustre: *** cfs_fail_loc=135, val=0*** [ 2416.209351] LustreError: 71795:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2416.218308] LustreError: 71795:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.133@tcp: cfg command failed: rc = -2 [ 2416.223884] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2416.223884] [ 2416.248760] LustreError: MGC192.168.202.133@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2416.257828] LustreError: 71783:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2416.273649] LustreError: 71783:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 2416.287395] LustreError: 71783:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 2416.304430] LustreError: 71783:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2416.326989] Lustre: server umount lustre-MDT0000 complete [ 2416.332895] LustreError: 71783:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2423.464070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2423.604719] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2423.851056] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2423.922398] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2427.717632] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2428.750565] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2428.920595] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2428.959239] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2429.032293] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2439.139547] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2439.170768] Lustre: Skipped 1 previous similar message [ 2439.185761] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2442.871750] Lustre: server umount lustre-OST0000 complete [ 2449.387909] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2449.398922] Lustre: Skipped 2 previous similar messages [ 2449.399641] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2449.402879] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2449.415662] Lustre: Skipped 2 previous similar messages [ 2454.507636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2454.517691] Lustre: Skipped 1 previous similar message [ 2459.618719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2461.152225] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2461.487183] Lustre: server umount lustre-MDT0000 complete [ 2464.741476] LustreError: 68705:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2464.777754] LustreError: 68705:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 2465.457325] LustreError: 69440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788853125 with bad export cookie 12014233740923146734 [ 2465.465454] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2465.467469] LustreError: 69440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2465.812738] Lustre: server umount lustre-MDT0001 complete [ 2476.200641] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 2479.740542] Key type lgssc unregistered [ 2480.006818] LNet: 73574:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2480.016423] LNetError: 73574:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2480.053746] LNet: Removed LNI 192.168.202.133@tcp [ 2481.353462] Key type .llcrypt unregistered [ 2481.355686] Key type ._llcrypt unregistered [ 2513.367991] Key type ._llcrypt registered [ 2513.371889] Key type .llcrypt registered [ 2513.548638] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 2515.851972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2516.023841] alg: No test for adler32 (adler32-zlib) [ 2517.486993] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 2517.919274] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 2519.664245] Key type lgssc registered [ 2522.018948] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2531.277920] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2531.321115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2532.900546] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2537.717853] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2543.087636] LustreError: 74986:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2543.126279] LustreError: 74986:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2546.530520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2547.002302] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2552.844952] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2562.021793] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2568.282730] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2575.519072] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2575.865145] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2577.945709] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2582.526879] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2589.908616] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2599.897318] Lustre: Failing over lustre-MDT0000 [ 2600.301215] Lustre: server umount lustre-MDT0000 complete [ 2602.358029] LustreError: 74983:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2602.470802] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2602.479810] Lustre: Skipped 1 previous similar message [ 2607.454448] LustreError: 76294:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2607.500197] LustreError: 76294:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2611.861415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2611.941624] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2612.084237] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2612.184425] LustreError: 78074:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2612.190756] LustreError: 78074:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.133@tcp: cfg command failed: rc = -9 [ 2612.199782] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2612.199782] [ 2612.207090] LustreError: MGC192.168.202.133@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2612.214693] LustreError: 78062:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2612.219385] LustreError: 78062:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -9 [ 2612.222900] LustreError: 78062:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 2612.226944] LustreError: 78062:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2612.241806] Lustre: server umount lustre-MDT0000 complete [ 2612.244353] LustreError: 78062:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2612.556699] LustreError: 74983:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2612.593789] LustreError: 74983:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2617.683139] LustreError: 76294:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2617.707407] LustreError: 76294:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2620.849932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2620.998145] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2621.008992] LustreError: 74979:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2621.313486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2621.377605] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2622.561505] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2626.564150] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2626.628148] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2626.760101] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2627.526936] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2638.307028] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2638.337878] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2638.381133] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2641.892253] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2641.899618] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2643.767524] Lustre: server umount lustre-OST0000 complete [ 2652.133508] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2652.167838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2652.187844] Lustre: Skipped 2 previous similar messages [ 2657.264619] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2657.287025] Lustre: Skipped 1 previous similar message [ 2657.577322] Lustre: server umount lustre-MDT0000 complete [ 2662.374206] LustreError: 75714:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2662.400125] LustreError: 75714:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 2662.491157] LustreError: 74967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788853322 with bad export cookie 4243159399084029798 [ 2662.493391] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2662.504627] LustreError: 74967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2662.934398] Lustre: server umount lustre-MDT0001 complete [ 2675.378558] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 2678.306820] Key type lgssc unregistered [ 2678.757815] LNet: 79857:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2678.766606] LNetError: 79857:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2678.782768] LNet: Removed LNI 192.168.202.133@tcp [ 2679.578857] Key type .llcrypt unregistered [ 2679.580835] Key type ._llcrypt unregistered [ 2694.737758] Key type ._llcrypt registered [ 2694.741693] Key type .llcrypt registered [ 2694.876405] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 2695.364704] Key type .llcrypt unregistered [ 2695.372207] Key type ._llcrypt unregistered [ 2722.440601] Key type ._llcrypt registered [ 2722.445499] Key type .llcrypt registered [ 2722.614433] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 2723.371202] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2723.409024] alg: No test for adler32 (adler32-zlib) [ 2724.630393] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 2724.861234] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 2726.488325] Key type lgssc registered [ 2727.788749] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2735.615473] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2735.646434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2737.480142] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2742.457234] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2746.850898] LustreError: 81752:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2746.869306] LustreError: 81752:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2751.311447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2751.662262] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2757.079882] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2765.277585] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2771.045142] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2777.934233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2778.479042] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2782.591186] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2785.724659] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2792.470263] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2806.245495] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2806.255689] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2806.268561] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2808.272990] Lustre: server umount lustre-OST0000 complete [ 2816.483441] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2816.509608] Lustre: Skipped 1 previous similar message [ 2816.526844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2816.540198] Lustre: Skipped 2 previous similar messages [ 2820.513330] Lustre: server umount lustre-MDT0000 complete [ 2821.604075] LustreError: 81753:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2821.626971] LustreError: 81753:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2825.312050] LustreError: 81734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788853485 with bad export cookie 13475644586317967700 [ 2825.314348] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2825.330318] LustreError: 81734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2825.687709] Lustre: server umount lustre-MDT0001 complete [ 2834.697581] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 2838.463552] Key type lgssc unregistered [ 2838.954844] LNet: 85304:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2838.971043] LNetError: 85304:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2838.999926] LNet: Removed LNI 192.168.202.133@tcp [ 2840.385161] Key type .llcrypt unregistered [ 2840.397708] Key type ._llcrypt unregistered [ 2865.005195] Key type ._llcrypt registered [ 2865.008479] Key type .llcrypt registered [ 2865.137590] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 2866.469985] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2866.609930] alg: No test for adler32 (adler32-zlib) [ 2867.754636] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 2868.131549] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 2869.840213] Key type lgssc registered [ 2871.187546] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2879.475258] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2879.492178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2881.008945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2885.808775] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2891.236622] LustreError: 86712:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2891.247831] LustreError: 86712:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2894.823890] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2895.276450] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2900.387219] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2908.237962] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2913.586392] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2920.996564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2921.480551] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2926.584885] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 2926.719327] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2937.308142] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2947.045422] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2947.050314] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2947.068140] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2950.116943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2950.126243] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2950.138388] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2951.559422] Lustre: server umount lustre-OST0000 complete [ 2955.238967] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2955.251124] Lustre: Skipped 1 previous similar message [ 2955.256455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2955.266870] Lustre: Skipped 2 previous similar messages [ 2960.353886] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2960.356307] Lustre: Skipped 1 previous similar message [ 2960.634943] Lustre: server umount lustre-MDT0000 complete [ 2963.863570] LustreError: 86694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788853624 with bad export cookie 14530669509318340493 [ 2963.872297] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2963.903817] LustreError: 86694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2964.232940] Lustre: server umount lustre-MDT0001 complete [ 2978.973342] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 2981.705218] Key type lgssc unregistered [ 2981.947849] LNet: 90243:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2981.952560] LNetError: 90243:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2981.972791] LNet: Removed LNI 192.168.202.133@tcp [ 2982.837141] Key type .llcrypt unregistered [ 2982.840839] Key type ._llcrypt unregistered [ 3005.041620] Key type ._llcrypt registered [ 3005.047487] Key type .llcrypt registered [ 3005.227936] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 3006.575683] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3006.631203] alg: No test for adler32 (adler32-zlib) [ 3007.784938] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 3008.100213] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 3009.796429] Key type lgssc registered [ 3011.352809] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3020.240792] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3020.260337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3021.749404] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3026.513249] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3031.016264] LustreError: 91640:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3031.060108] LustreError: 91640:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3034.537185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3034.853941] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3041.059687] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3048.190958] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3053.600712] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3059.809082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3060.246952] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3066.417748] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 3067.941893] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3076.256080] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3088.883429] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3088.890446] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3088.934710] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3091.939556] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3091.943626] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3091.946765] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3091.959495] Lustre: Skipped 1 previous similar message [ 3093.680863] Lustre: server umount lustre-OST0000 complete [ 3101.665595] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3101.669137] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3101.678604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3104.417772] Lustre: server umount lustre-MDT0000 complete [ 3107.300381] LustreError: 91641:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3107.378660] LustreError: 91618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788853767 with bad export cookie 7563613825070313458 [ 3107.389927] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3107.392645] LustreError: 91618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3107.633455] Lustre: server umount lustre-MDT0001 complete [ 3119.016602] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 3122.538370] Key type lgssc unregistered [ 3122.930680] LNet: 95174:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3122.953485] LNetError: 95174:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3124.007818] LNet: Removed LNI 192.168.202.133@tcp [ 3125.482346] Key type .llcrypt unregistered [ 3125.483825] Key type ._llcrypt unregistered [ 3148.758369] Key type ._llcrypt registered [ 3148.760653] Key type .llcrypt registered [ 3148.845628] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 3149.956768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3149.980296] alg: No test for adler32 (adler32-zlib) [ 3151.062725] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 3151.307810] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 3152.970418] Key type lgssc registered [ 3154.990085] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3164.895449] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3164.913836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3180.448643] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3185.568400] LustreError: 96556:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.133@tcp: failed processing log, type 4: rc = -110 [ 3212.256235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3216.823060] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3223.705702] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3229.037318] Lustre: server umount lustre-OST0000 complete [ 3236.154661] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 03:51:34 (1788853894) [ 3243.472700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3244.061062] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3249.128944] LustreError: 98196:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3249.161711] LustreError: 98196:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 3250.226826] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3254.242447] LustreError: 98197:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3255.270465] LustreError: 98196:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3260.405209] LustreError: 98197:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3261.862839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3262.457897] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3267.220764] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3275.295357] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3281.850950] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3288.314938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3288.763484] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3289.804485] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 3295.336967] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3303.401335] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3317.735280] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3317.741406] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3317.741891] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3317.746790] Lustre: Skipped 1 previous similar message [ 3319.695190] Lustre: server umount lustre-OST0000 complete [ 3327.976751] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3327.996052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3328.011123] Lustre: Skipped 1 previous similar message [ 3328.039966] Lustre: Skipped 2 previous similar messages [ 3330.279053] Lustre: server umount lustre-MDT0000 complete [ 3333.090202] LustreError: 99194:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3333.131536] LustreError: 99194:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3334.297121] LustreError: 98178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788853994 with bad export cookie 12648803617228506131 [ 3334.315259] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3334.607654] Lustre: server umount lustre-MDT0001 complete [ 3346.747790] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 3349.021970] Key type lgssc unregistered [ 3349.340737] LNet: 101887:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3349.348176] LNetError: 101887:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3349.366478] LNet: Removed LNI 192.168.202.133@tcp [ 3350.247250] Key type .llcrypt unregistered [ 3350.249158] Key type ._llcrypt unregistered [ 3393.950479] Key type ._llcrypt registered [ 3393.956320] Key type .llcrypt registered [ 3394.134988] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 3396.237971] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3396.372258] alg: No test for adler32 (adler32-zlib) [ 3397.615746] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 3398.045991] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 3399.939480] Key type lgssc registered [ 3401.470955] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3409.193516] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3409.207629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3410.638123] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3414.826067] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3420.137306] LustreError: 104070:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3420.151293] LustreError: 104070:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3423.296806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3423.560400] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3428.455776] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3436.738996] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3444.383419] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3450.684101] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3450.971060] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3455.086804] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 3457.542498] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3464.649871] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3478.008521] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3478.011083] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3478.018039] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3479.011473] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3479.018465] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3479.021507] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3482.446559] Lustre: server umount lustre-OST0000 complete [ 3489.260583] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3489.266911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3489.285397] Lustre: Skipped 1 previous similar message [ 3489.292018] Lustre: Skipped 1 previous similar message [ 3492.647986] Lustre: server umount lustre-MDT0000 complete [ 3494.372445] LustreError: 104070:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3494.395514] LustreError: 104070:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3495.907345] LustreError: 104049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788854156 with bad export cookie 17835421003911857300 [ 3495.916359] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3495.919171] LustreError: 104049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3496.261525] Lustre: server umount lustre-MDT0001 complete [ 3506.336913] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 3508.811576] Key type lgssc unregistered [ 3509.140658] LNet: 107603:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3509.163361] LNetError: 107603:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3509.185524] LNet: Removed LNI 192.168.202.133@tcp [ 3510.347872] Key type .llcrypt unregistered [ 3510.363920] Key type ._llcrypt unregistered [ 3532.653487] Key type ._llcrypt registered [ 3532.655392] Key type .llcrypt registered [ 3532.763920] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 3533.784592] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3533.828084] alg: No test for adler32 (adler32-zlib) [ 3534.845312] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 3534.993859] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 3536.632234] Key type lgssc registered [ 3537.743606] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3549.614364] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3549.645096] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3565.408507] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3570.528382] LustreError: 108842:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.133@tcp: failed processing log, type 4: rc = -110 [ 3597.220383] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3603.939807] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3612.320460] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3619.214437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3619.454629] LustreError: MGC192.168.202.133@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3619.460687] LustreError: 109867:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3619.463376] LustreError: 109867:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 3619.466357] LustreError: 109867:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 3619.529880] Lustre: server umount lustre-MDT0000 complete [ 3619.539757] LustreError: 109867:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3639.776592] Lustre: 108178:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788854284/real 1788854284] req@00000000b588715a x1875749576379648/t0(0) o400->MGC192.168.202.133@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788854300 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3639.831807] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3639.857194] LustreError: 108854:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3640.162758] Lustre: server umount lustre-OST0000 complete [ 3652.405284] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 3661.825886] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 3702.102790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3702.369395] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3702.422737] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3702.535706] Lustre: lustre-MDT0000: new disk, initializing [ 3702.714879] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3702.734472] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3707.636052] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3718.785218] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3718.905099] Lustre: 112661:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3718.971691] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3718.976211] Lustre: Skipped 1 previous similar message [ 3719.039534] Lustre: lustre-MDT0001: new disk, initializing [ 3719.171246] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3719.223956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3719.242710] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3723.217969] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3731.997638] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3737.925458] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3744.415806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3744.711828] Lustre: lustre-OST0000: new disk, initializing [ 3744.715307] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3744.726573] Lustre: 114627:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3744.863414] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3751.425136] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3752.046190] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3752.061733] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3752.188733] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3762.268260] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3768.458976] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3768.907825] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3774.018778] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3774.171936] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3778.018252] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3778.037240] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3778.066446] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3782.470310] Lustre: server umount lustre-OST0000 complete [ 3788.263698] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3788.274087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3788.295078] Lustre: Skipped 2 previous similar messages [ 3788.296628] Lustre: Skipped 1 previous similar message [ 3793.377535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3793.379837] Lustre: Skipped 1 previous similar message [ 3798.503419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3798.510102] Lustre: Skipped 1 previous similar message [ 3801.058520] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3801.260932] Lustre: server umount lustre-MDT0000 complete [ 3805.160870] LustreError: 112653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788854465 with bad export cookie 6965416441628722837 [ 3805.178160] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3805.182758] LustreError: 112653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3805.760720] Lustre: server umount lustre-MDT0001 complete [ 3815.786932] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 04:01:14 (1788854474) [ 3818.659113] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3837.062310] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 3849.638108] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 3889.046587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3889.351369] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3889.372480] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3889.447821] Lustre: lustre-MDT0000: new disk, initializing [ 3889.536641] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3889.594100] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3894.095738] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3905.827210] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3906.052493] Lustre: 119542:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3906.132879] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3906.142101] Lustre: Skipped 1 previous similar message [ 3906.359727] Lustre: lustre-MDT0001: new disk, initializing [ 3906.564961] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3906.640749] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3906.667579] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3913.318371] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3924.368066] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3931.808616] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3939.226611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3939.635225] Lustre: lustre-OST0000: new disk, initializing [ 3939.639984] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3939.654737] Lustre: 121511:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3939.787956] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3947.547262] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3948.595982] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3948.605729] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3948.734655] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3958.167759] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3963.439507] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3963.778461] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3969.694754] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3969.973746] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3974.632795] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3974.643996] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3974.663421] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3974.666334] LustreError: Skipped 1 previous similar message [ 3974.681292] Lustre: Skipped 1 previous similar message [ 3984.882580] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3984.898849] Lustre: Skipped 3 previous similar messages [ 3986.403412] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3986.697137] Lustre: server umount lustre-OST0000 complete [ 3993.575525] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3993.589666] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3993.614341] Lustre: Skipped 1 previous similar message [ 4005.344165] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4005.349115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4005.383792] Lustre: Skipped 6 previous similar messages [ 4005.834701] Lustre: server umount lustre-MDT0000 complete [ 4010.207366] LustreError: 120345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788854670 with bad export cookie 6965416441628723719 [ 4010.210617] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4010.215396] LustreError: 120345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4010.797729] Lustre: server umount lustre-MDT0001 complete [ 4020.551886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4020.888870] LustreError: 123730:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4020.902784] LustreError: 123730:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4020.944681] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4025.117210] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4026.343052] LustreError: 123731:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4031.459724] LustreError: 123730:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4036.580403] LustreError: 123731:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4036.782314] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4042.533348] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4052.869804] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4059.310132] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4067.897781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4074.822627] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4081.967512] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4099.734973] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 4106.398961] Lustre: server umount lustre-OST0000 complete [ 4112.353750] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4112.363357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4112.378787] Lustre: Skipped 2 previous similar messages [ 4120.899841] Lustre: server umount lustre-MDT0000 complete [ 4122.596477] LustreError: 123730:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4122.619798] LustreError: 123730:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4123.617107] Lustre: 108178:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788854767/real 1788854767] req@0000000085bb05e1 x1875749576500224/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788854783 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4123.673753] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4124.710536] Lustre: 108180:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788854768/real 1788854768] req@0000000026823f73 x1875749576500480/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788854784 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4125.689738] LustreError: 123712:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788854785 with bad export cookie 6965416441628724601 [ 4125.700824] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4125.720095] LustreError: 123712:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4126.261537] Lustre: server umount lustre-MDT0001 complete [ 4138.499489] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 4141.267697] Key type lgssc unregistered [ 4141.556279] LNet: 127381:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4141.563033] LNetError: 127381:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4142.635311] LNet: Removed LNI 192.168.202.133@tcp [ 4143.433444] Key type .llcrypt unregistered [ 4143.436235] Key type ._llcrypt unregistered [ 4167.178798] Key type ._llcrypt registered [ 4167.181058] Key type .llcrypt registered [ 4167.283432] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 4185.060898] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 4186.679706] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4186.802383] alg: No test for adler32 (adler32-zlib) [ 4188.238347] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 4188.581751] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 4190.490687] Key type lgssc registered [ 4191.966701] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4232.629514] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4232.663277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4234.050704] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4234.086439] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4234.249436] Lustre: lustre-MDT0000: new disk, initializing [ 4234.383946] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4234.404869] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4240.082694] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4251.168378] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4251.273437] Lustre: 130534:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4251.357529] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4251.361834] Lustre: Skipped 1 previous similar message [ 4251.424774] Lustre: lustre-MDT0001: new disk, initializing [ 4251.543231] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4251.599327] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4251.617080] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4255.785490] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4264.809946] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4271.102294] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4279.116489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4279.341322] Lustre: lustre-OST0000: new disk, initializing [ 4279.344339] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4279.349698] Lustre: 132505:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4279.444630] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4284.796319] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4287.049810] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4287.058991] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4287.110891] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4294.573352] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4299.603889] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4299.835806] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4304.789615] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4304.991876] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4307.939320] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4307.952246] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4307.953618] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4313.069938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4313.090972] Lustre: Skipped 2 previous similar messages [ 4318.178071] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4321.248244] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4321.609800] Lustre: server umount lustre-OST0000 complete [ 4328.420037] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4328.421825] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4328.427379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4328.443107] Lustre: Skipped 2 previous similar messages [ 4328.449363] Lustre: Skipped 1 previous similar message [ 4332.040379] Lustre: server umount lustre-MDT0000 complete [ 4336.402331] LustreError: 130527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788854996 with bad export cookie 690070672910303425 [ 4336.410504] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4336.426221] LustreError: 130527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4336.986289] Lustre: server umount lustre-MDT0001 complete [ 4346.453800] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 04:10:05 (1788855005) [ 4358.458873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4358.965996] LustreError: 134919:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4359.012712] LustreError: 134919:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4359.124195] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4364.259392] LustreError: 134920:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4364.302599] LustreError: 134920:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4365.730613] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4369.377295] LustreError: 134919:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4374.498499] LustreError: 134920:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4374.966179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4375.366476] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4380.506912] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4389.831778] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4397.944841] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4400.867656] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4400.883890] Lustre: Skipped 1 previous similar message [ 4400.884660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4400.899432] Lustre: Skipped 1 previous similar message [ 4406.089354] LustreError: 136489:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x2:0x0].0x0 (000000000b9c169a) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 4406.214619] Lustre: server umount lustre-MDT0000 complete [ 4406.242635] LustreError: 135647:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4406.250427] LustreError: 135647:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4411.298551] LustreError: 134899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788855071 with bad export cookie 690070672910304307 [ 4411.306650] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4411.587254] Lustre: server umount lustre-MDT0001 complete [ 4422.861053] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 04:11:20 (1788855080) [ 4431.515407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4447.138065] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4452.256532] LustreError: 137336:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.133@tcp: failed processing log, type 4: rc = -110 [ 4477.985507] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4485.026143] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4494.144845] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4496.662511] Lustre: server umount lustre-OST0000 complete [ 4507.452813] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 04:12:45 (1788855165) [ 4515.753439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4516.297315] LustreError: 138784:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4516.425836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4521.345315] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4530.709721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4535.494689] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4544.902798] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4551.985719] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4560.537169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4560.884264] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 4561.057580] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4561.078069] Lustre: Skipped 1 previous similar message [ 4569.800702] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4579.983080] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4586.978849] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4586.983355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4587.000530] Lustre: Skipped 1 previous similar message [ 4593.142747] Lustre: server umount lustre-MDT0000 complete [ 4597.230342] LustreError: 138779:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4597.259610] LustreError: 138779:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 4598.212571] LustreError: 138766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788855258 with bad export cookie 690070672910304972 [ 4598.222053] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4598.235516] LustreError: 138766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 4599.110576] Lustre: server umount lustre-MDT0001 complete [ 4614.873207] Lustre: server umount lustre-OST0000 complete [ 4624.555310] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 04:14:43 (1788855283) [ 4632.186960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4632.749381] LustreError: 142439:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4632.878231] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4639.674332] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4650.650241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4656.402700] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4665.249368] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4671.610689] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4679.016725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4686.277124] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4695.898233] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4702.929308] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4703.658142] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4704.059082] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4710.623986] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4710.877994] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4713.443337] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4713.446969] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4713.469387] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4713.482900] Lustre: Skipped 2 previous similar messages [ 4713.491878] Lustre: Skipped 6 previous similar messages [ 4727.264189] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4727.575345] Lustre: server umount lustre-OST0000 complete [ 4732.901812] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4732.920737] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4746.208316] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4746.439850] Lustre: server umount lustre-MDT0000 complete [ 4749.287634] LustreError: 142436:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4749.318432] LustreError: 142436:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 4750.982747] LustreError: 142420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788855411 with bad export cookie 690070672910306043 [ 4750.991333] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4751.001819] LustreError: 142420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4751.027924] LustreError: 128755:0:(import.c:762:ptlrpc_connect_import_locked()) already connecting [ 4751.425946] Lustre: server umount lustre-MDT0001 complete [ 4761.634991] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 04:17:00 (1788855420) [ 4769.449203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4785.184610] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4790.304625] LustreError: 146603:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.133@tcp: failed processing log, type 4: rc = -110 [ 4816.160286] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4816.174887] Lustre: Skipped 2 previous similar messages [ 4825.148713] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4834.664187] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4843.388041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4844.444451] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4850.159529] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4860.818674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4864.854603] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4872.220190] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4880.453376] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4886.406948] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4886.667816] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4892.737844] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4893.059125] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4897.267029] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4897.269607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4897.269714] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4897.277042] Lustre: Skipped 3 previous similar messages [ 4897.317129] Lustre: Skipped 13 previous similar messages [ 4901.561407] Lustre: server umount lustre-MDT0000 complete [ 4902.376394] LustreError: 147643:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4902.412533] LustreError: 147643:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 4905.751487] LustreError: 146609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788855566 with bad export cookie 690070672910307065 [ 4905.759839] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4905.760537] LustreError: 146609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4906.293826] Lustre: server umount lustre-MDT0001 complete [ 4920.131448] Lustre: server umount lustre-OST0000 complete [ 4929.935700] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 04:19:48 (1788855588) [ 4937.712562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4953.376947] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4958.496816] LustreError: 150802:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.133@tcp: failed processing log, type 4: rc = -110 [ 4990.902506] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4999.307715] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5007.048046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5007.443411] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 5012.260436] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5021.163947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5027.208695] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5034.467486] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5041.325846] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5047.884315] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5048.205209] Lustre: lustre-OST0001: new disk, initializing [ 5048.211929] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5048.217338] Lustre: 153677:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5056.900685] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5058.253140] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5058.263599] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5058.414459] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5066.344834] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5071.792102] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5072.044479] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5078.178503] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5078.496946] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5083.105827] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5083.116316] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5083.132599] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5095.392606] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5095.587486] Lustre: server umount lustre-OST0000 complete [ 5104.617249] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5104.641694] LustreError: Skipped 2 previous similar messages [ 5115.360801] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5115.756242] Lustre: server umount lustre-OST0001 complete [ 5126.627703] Lustre: server umount lustre-MDT0000 complete [ 5131.242362] LustreError: 150810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788855791 with bad export cookie 690070672910308017 [ 5131.251835] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5131.252902] LustreError: 150810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5131.695573] Lustre: server umount lustre-MDT0001 complete [ 5157.062486] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 04:23:36 (1788855816) [ 5158.671120] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 5160.208540] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 5161.616868] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 04:23:40 (1788855820) [ 5168.875400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5169.315677] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5169.342949] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5169.648801] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5169.664150] Lustre: Skipped 6 previous similar messages [ 5174.252539] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5183.053500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5183.234281] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5183.289280] Lustre: 157239:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5188.792817] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5197.647810] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5205.002468] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5212.777130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5213.067049] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5219.105626] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 5221.131302] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5229.350418] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5235.682770] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5236.024269] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5241.822700] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5242.126591] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5244.898337] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5244.908174] LustreError: Skipped 1 previous similar message [ 5244.916230] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5244.934330] Lustre: Skipped 5 previous similar messages [ 5250.513771] Lustre: server umount lustre-OST0000 complete [ 5274.100886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5275.442352] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5275.478300] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5275.479068] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5282.530507] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5291.749442] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5297.518268] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5297.855556] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5301.716361] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5301.933642] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5308.685542] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffffa00145a0a800.ost_server_uuid 50 [ 5310.407343] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffffa00145a0a800.ost_server_uuid in FULL state after 0 sec [ 5310.691317] LustreError: 160506:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.202.33@tcp) returned error from blocking AST (req@000000004e351886 x1875750261378304 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000f2fb6c9a/0x9939f9ef0bbdc0b lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.33@tcp remote: 0x4034d4936eb1013a expref: 5 pid: 157248 timeout: 5410 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5310.764741] LustreError: lustre-MDT0000: A client on nid 192.168.202.33@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5310.795549] LustreError: 157237:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.202.33@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000f2fb6c9a/0x9939f9ef0bbdc0b lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.33@tcp remote: 0x4034d4936eb1013a expref: 6 pid: 157248 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5316.583056] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5316.607016] LustreError: Skipped 2 previous similar messages [ 5321.727695] LustreError: 160869:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5321.760156] LustreError: 160869:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 16 previous similar messages [ 5321.968348] Lustre: server umount lustre-OST0000 complete [ 5341.152333] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5341.483446] Lustre: server umount lustre-MDT0000 complete [ 5345.866057] LustreError: 157234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788856006 with bad export cookie 690070672910309200 [ 5345.874585] LustreError: 157234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5345.887854] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5352.178938] Lustre: server umount lustre-MDT0001 complete [ 5367.153247] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 5372.001191] Key type lgssc unregistered [ 5372.489613] LNet: 163357:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5372.495310] LNetError: 163357:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5373.550160] LNet: Removed LNI 192.168.202.133@tcp [ 5374.900634] Key type .llcrypt unregistered [ 5374.907096] Key type ._llcrypt unregistered [ 5400.937160] Key type ._llcrypt registered [ 5400.945577] Key type .llcrypt registered [ 5401.124876] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 5402.655290] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5402.779750] alg: No test for adler32 (adler32-zlib) [ 5404.091619] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 5404.309870] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 5405.984269] Key type lgssc registered [ 5407.428884] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5416.455254] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5416.475844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5418.089711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5422.575355] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5428.716600] LustreError: 164767:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5428.734542] LustreError: 164767:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5431.477263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5431.974881] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5436.855628] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5446.088379] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5452.261854] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5459.967178] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5460.517316] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5463.618303] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5468.004101] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5475.515370] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5480.469335] Lustre: Failing over lustre-MDT0000 [ 5480.691722] Lustre: server umount lustre-MDT0000 complete [ 5481.954677] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5481.960405] Lustre: Skipped 2 previous similar messages [ 5481.967277] LustreError: 164768:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5482.000101] LustreError: 164768:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5487.075266] LustreError: 164767:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5487.104674] LustreError: 164767:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 5492.194456] LustreError: 165496:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5492.227810] LustreError: 165496:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5493.279638] LustreError: 164748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.33@tcp arrived at 1788856153 with bad export cookie 4753462022140028174 [ 5497.313632] LustreError: 164768:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5497.347026] LustreError: 164768:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5498.346678] Lustre: 164078:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788856142/real 1788856142] req@00000000227d9757 x1875751536191232/t0(0) o400->MGC192.168.202.133@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788856158 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5498.411832] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5504.065478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5508.582625] LustreError: 164767:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5508.621083] LustreError: 164767:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 5509.052807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5509.117256] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5514.217288] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5514.228087] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5516.051777] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5519.915254] Lustre: lustre-MDT0000: Denying connection for new client c4db279d-1fd6-429c-ad4f-473ff9419ff4 (at 192.168.202.33@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:04 [ 5529.571742] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5529.606026] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5529.615392] Lustre: Skipped 1 previous similar message [ 5534.693931] LustreError: 166588:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5534.735817] LustreError: 166588:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5534.859468] Lustre: server umount lustre-OST0000 complete [ 5539.432969] LustreError: 168499:0:(ldlm_lib.c:3038:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5539.439139] Lustre: 167845:0:(ldlm_lib.c:2438:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5539.443863] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5539.461816] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5539.468609] LustreError: 167845:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000052c708d0 x1875751536212992/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5539.476607] LustreError: 167845:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5539.479616] LustreError: 167845:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5539.483068] Lustre: lustre-MDT0000: Recovery over after 0:25, of 2 clients 0 recovered and 2 were evicted. [ 5539.810954] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5539.831338] Lustre: Skipped 1 previous similar message [ 5541.743116] Lustre: server umount lustre-MDT0000 complete [ 5546.448828] LustreError: 168297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788856206 with bad export cookie 4753462022140028279 [ 5546.450413] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5546.460683] LustreError: 168297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 5546.474307] LustreError: 164076:0:(import.c:752:ptlrpc_connect_import_locked()) can't connect to a closed import [ 5553.095043] Lustre: server umount lustre-MDT0001 complete [ 5569.933511] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 5574.799304] Key type lgssc unregistered [ 5575.404530] LNet: 169253:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5575.407724] LNetError: 169253:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5575.429637] LNet: Removed LNI 192.168.202.133@tcp [ 5576.611165] Key type .llcrypt unregistered [ 5576.621834] Key type ._llcrypt unregistered [ 5606.355835] Key type ._llcrypt registered [ 5606.358558] Key type .llcrypt registered [ 5606.471350] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 5607.753356] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5607.980074] alg: No test for adler32 (adler32-zlib) [ 5609.067179] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 5609.324420] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 5611.024232] Key type lgssc registered [ 5612.619602] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5621.893259] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5621.916465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5623.554212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5629.201974] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5632.997045] LustreError: 170662:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5633.013568] LustreError: 170662:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5638.117121] LustreError: 170663:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5639.282470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5639.994892] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5645.189788] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5654.031189] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5659.940385] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5667.854706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5668.565079] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5676.045180] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5683.534629] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5693.123959] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5700.577546] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5700.580079] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5700.585836] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5703.652601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5703.654642] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5703.674946] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5708.771665] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5708.778730] Lustre: Skipped 1 previous similar message [ 5713.889875] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5713.897123] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5713.902772] Lustre: Skipped 1 previous similar message [ 5714.010843] Lustre: server umount lustre-OST0000 complete [ 5719.013116] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5719.028472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5719.029643] Lustre: Skipped 1 previous similar message [ 5719.064302] Lustre: Skipped 2 previous similar messages [ 5729.249731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5729.252702] Lustre: Skipped 3 previous similar messages [ 5732.320270] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5732.522519] Lustre: server umount lustre-MDT0000 complete [ 5734.373127] LustreError: 171650:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5734.402482] LustreError: 171650:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5736.577980] LustreError: 170642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788856396 with bad export cookie 2697934212586719493 [ 5736.585305] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5736.922573] Lustre: server umount lustre-MDT0001 complete [ 5750.481956] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 5753.601333] Key type lgssc unregistered [ 5753.937707] LNet: 174219:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5753.943233] LNetError: 174219:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5753.951795] LNet: Removed LNI 192.168.202.133@tcp [ 5754.512735] Key type .llcrypt unregistered [ 5754.514337] Key type ._llcrypt unregistered [ 5789.637734] Key type ._llcrypt registered [ 5789.642391] Key type .llcrypt registered [ 5789.788248] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 5790.571879] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5790.586867] alg: No test for adler32 (adler32-zlib) [ 5791.837664] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 5792.223496] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 5793.896229] Key type lgssc registered [ 5795.904406] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5808.205357] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5808.239882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5809.940392] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5816.591125] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5820.393759] LustreError: 175918:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5820.413491] LustreError: 175918:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5825.513798] LustreError: 175919:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5827.379024] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5827.731770] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5833.518776] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5841.637638] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5847.246912] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5855.825917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5856.391436] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5864.645348] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5874.454557] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5879.915900] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5888.482721] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5888.492747] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5888.517149] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5890.551038] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5890.555824] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5890.560286] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5890.612035] Lustre: Skipped 1 previous similar message [ 5892.606376] Lustre: server umount lustre-OST0000 complete [ 5899.234266] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5899.256481] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5899.293957] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5905.504601] Lustre: server umount lustre-MDT0000 complete [ 5905.892628] LustreError: 175918:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5905.911983] LustreError: 175918:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5910.718570] LustreError: 175899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788856570 with bad export cookie 12639634184986473964 [ 5910.734186] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5910.749907] LustreError: 175899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5911.469297] Lustre: server umount lustre-MDT0001 complete [ 5926.949108] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 5930.488478] Key type lgssc unregistered [ 5930.925868] LNet: 179454:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5930.946302] LNetError: 179454:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5930.968403] LNet: Removed LNI 192.168.202.133@tcp [ 5932.383823] Key type .llcrypt unregistered [ 5932.395611] Key type ._llcrypt unregistered [ 5959.122749] Key type ._llcrypt registered [ 5959.133092] Key type .llcrypt registered [ 5959.333240] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 5960.836327] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5960.892341] alg: No test for adler32 (adler32-zlib) [ 5961.983541] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 5962.293745] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 5963.968511] Key type lgssc registered [ 5965.492328] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5979.151649] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5979.172264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5980.713188] Lustre: *** cfs_fail_loc=135, val=0*** [ 5980.979282] LustreError: 180977:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5980.995550] LustreError: 180977:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.133@tcp: cfg command failed: rc = -2 [ 5981.016289] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5981.016289] [ 5981.033692] LustreError: MGC192.168.202.133@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5981.056439] LustreError: 180939:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5981.071900] LustreError: 180939:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 5981.083494] LustreError: 180939:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 5981.099832] LustreError: 180939:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5981.279063] Lustre: server umount lustre-MDT0000 complete [ 5981.283257] LustreError: 180939:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5991.181922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5991.724109] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5998.075638] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6001.141431] LustreError: 181349:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6001.163329] LustreError: 181349:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6006.246253] LustreError: 181350:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6006.884098] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6007.409091] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6012.884842] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6021.090606] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6027.092928] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6032.353717] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6032.381977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6033.444369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6033.457728] Lustre: Skipped 2 previous similar messages [ 6035.899300] Lustre: server umount lustre-MDT0000 complete [ 6038.500468] LustreError: 181350:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6038.530956] LustreError: 181350:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6040.354487] LustreError: 181329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788856700 with bad export cookie 14338659969082279577 [ 6040.364275] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6040.373443] LustreError: 181329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6040.857481] Lustre: server umount lustre-MDT0001 complete [ 6054.318990] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 6058.177958] Key type lgssc unregistered [ 6058.532602] LNet: 183669:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6058.556468] LNetError: 183669:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6058.582799] LNet: Removed LNI 192.168.202.133@tcp [ 6059.616711] Key type .llcrypt unregistered [ 6059.619876] Key type ._llcrypt unregistered [ 6084.174334] Key type ._llcrypt registered [ 6084.175839] Key type .llcrypt registered [ 6084.348215] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 6084.679794] Key type .llcrypt unregistered [ 6084.681082] Key type ._llcrypt unregistered [ 6107.513758] Key type ._llcrypt registered [ 6107.518302] Key type .llcrypt registered [ 6107.655671] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 6108.813341] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6108.876073] alg: No test for adler32 (adler32-zlib) [ 6109.976248] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 6110.222108] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 6111.920162] Key type lgssc registered [ 6113.712755] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6124.160096] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6124.205624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6139.872556] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6144.992764] LustreError: 185797:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.133@tcp: failed processing log, type 4: rc = -110 [ 6171.616250] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6179.405892] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6188.402292] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6195.005858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6195.587821] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6195.656871] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 6200.801823] LustreError: 185823:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6200.838463] LustreError: 185823:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6200.924951] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6205.925545] LustreError: 185822:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6205.942511] LustreError: 185822:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6210.158716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6211.033047] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6216.838223] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6225.609681] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6230.706422] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6234.045635] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6246.894382] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6246.898463] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6246.905458] LustreError: Skipped 1 previous similar message [ 6246.920050] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6246.935246] Lustre: Skipped 1 previous similar message [ 6246.961570] Lustre: Skipped 1 previous similar message [ 6251.554557] Lustre: server umount lustre-OST0000 complete [ 6257.124617] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6257.125575] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6257.126490] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6257.138098] Lustre: Skipped 1 previous similar message [ 6261.545634] Lustre: server umount lustre-MDT0000 complete [ 6262.255306] LustreError: 186834:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6262.291643] LustreError: 186834:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6267.385251] LustreError: 186839:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6267.406894] LustreError: 186839:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6267.984161] LustreError: 185803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788856928 with bad export cookie 15925093984105432768 [ 6267.998465] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6268.004575] LustreError: 185803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6268.649875] Lustre: server umount lustre-MDT0001 complete [ 6285.582486] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 6289.327583] Key type lgssc unregistered [ 6289.643731] LNet: 189717:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6289.648545] LNetError: 189717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6289.676881] LNet: Removed LNI 192.168.202.133@tcp [ 6290.903388] Key type .llcrypt unregistered [ 6290.905067] Key type ._llcrypt unregistered [ 6317.847735] Key type ._llcrypt registered [ 6317.852665] Key type .llcrypt registered [ 6317.969542] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 6319.223081] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6319.293335] alg: No test for adler32 (adler32-zlib) [ 6320.379789] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 6320.731986] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 6322.576246] Key type lgssc registered [ 6324.226500] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6333.504610] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6333.520813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6335.132045] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6339.594802] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6345.699667] LustreError: 191127:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6345.734444] LustreError: 191127:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6350.487549] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6350.820997] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6351.060381] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6357.052180] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6367.008760] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6373.619134] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6381.051687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6381.443969] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6389.346427] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6398.631698] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6405.000683] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 6413.714740] Lustre: Failing over lustre-MDT0000 [ 6414.080365] Lustre: server umount lustre-MDT0000 complete [ 6415.228459] LustreError: 191123:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6415.333911] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6415.367510] Lustre: Skipped 2 previous similar messages [ 6420.293380] LustreError: 191124:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6420.324224] LustreError: 191124:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 6425.414784] LustreError: 191123:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6425.435444] LustreError: 191123:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 6430.541215] LustreError: 191122:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6430.568720] LustreError: 191122:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 6431.713034] Lustre: 190440:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788857075/real 1788857075] req@00000000c0fe547a x1875752496690560/t0(0) o400->MGC192.168.202.133@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788857091 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6431.745670] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6433.994735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6440.774840] LustreError: 191123:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6440.804333] LustreError: 191123:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 12 previous similar messages [ 6441.971890] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x877cb4016d1626aa [ 6441.980720] Lustre: MGC192.168.202.133@tcp: Connection restored to 0@lo (at 0@lo) [ 6442.194682] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6442.238745] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6445.891794] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6447.182647] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6447.611831] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6447.665128] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 6447.760443] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 6453.821819] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6456.690469] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6467.044955] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 6467.063406] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6467.090288] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6467.554636] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6468.928363] Lustre: server umount lustre-OST0000 complete [ 6473.190868] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6473.203632] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6473.207198] Lustre: Skipped 2 previous similar messages [ 6473.209723] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6473.209732] Lustre: Skipped 3 previous similar messages [ 6478.306199] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6478.316251] Lustre: Skipped 2 previous similar messages [ 6479.558754] Lustre: server umount lustre-MDT0000 complete [ 6483.427813] LustreError: 193620:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6483.457375] LustreError: 193620:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 6484.278455] LustreError: 194733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788857144 with bad export cookie 9762876010450527914 [ 6484.281604] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6484.306908] LustreError: 194733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6484.657924] Lustre: server umount lustre-MDT0001 complete [ 6499.051147] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 6503.122836] Key type lgssc unregistered [ 6503.447781] LNet: 195897:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6503.455103] LNetError: 195897:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6503.469312] LNet: Removed LNI 192.168.202.133@tcp [ 6504.758906] Key type .llcrypt unregistered [ 6504.764615] Key type ._llcrypt unregistered [ 6532.843695] Key type ._llcrypt registered [ 6532.845381] Key type .llcrypt registered [ 6533.002204] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 6534.738419] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6534.937080] alg: No test for adler32 (adler32-zlib) [ 6536.375729] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 6536.755079] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 6538.432289] Key type lgssc registered [ 6540.130417] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6550.354331] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6550.373088] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6551.985607] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6556.898432] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6557.158361] LustreError: 197311:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6557.183820] LustreError: 197311:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6562.289750] LustreError: 197312:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6567.323887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6567.402746] LustreError: 197311:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6567.929923] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6573.000199] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6584.133096] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6590.172174] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6597.905457] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6598.353189] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6607.137419] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6617.235058] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6621.975477] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6625.769911] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6629.275756] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6643.265668] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6652.938167] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6652.953490] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6652.955481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6652.984220] Lustre: Skipped 1 previous similar message [ 6656.208467] Lustre: server umount lustre-OST0000 complete [ 6663.141109] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6663.164418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6663.173789] Lustre: Skipped 1 previous similar message [ 6663.209911] Lustre: Skipped 2 previous similar messages [ 6668.270754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6668.849961] Lustre: server umount lustre-MDT0000 complete [ 6672.890985] LustreError: 197291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788857333 with bad export cookie 12163006581129528082 [ 6672.900424] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6672.912118] LustreError: 197291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6673.513338] Lustre: server umount lustre-MDT0001 complete [ 6686.657840] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 6689.911438] Key type lgssc unregistered [ 6690.257587] LNet: 201158:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6690.267363] LNetError: 201158:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6690.286328] LNet: Removed LNI 192.168.202.133@tcp [ 6691.411367] Key type .llcrypt unregistered [ 6691.417342] Key type ._llcrypt unregistered [ 6712.869845] Key type ._llcrypt registered [ 6712.872407] Key type .llcrypt registered [ 6712.941887] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 6713.708985] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6713.746586] alg: No test for adler32 (adler32-zlib) [ 6714.653453] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 6714.771897] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 6716.360347] Key type lgssc registered [ 6717.706602] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6726.552724] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6726.575654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6728.111415] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6733.796161] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6738.407715] LustreError: 202423:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6738.428887] LustreError: 202423:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6743.030718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6743.500571] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6748.257947] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6756.654671] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6763.550405] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6769.147695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6769.423511] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6773.560140] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6775.891835] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6786.000554] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6793.139432] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6805.397059] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6813.043995] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6819.187626] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6829.033368] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6829.040793] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6829.061532] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6829.067254] Lustre: Skipped 1 previous similar message [ 6833.285936] Lustre: server umount lustre-OST0000 complete [ 6839.273511] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6839.279717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6839.300790] Lustre: Skipped 2 previous similar messages [ 6839.313146] Lustre: Skipped 1 previous similar message [ 6843.754245] Lustre: server umount lustre-MDT0000 complete [ 6844.384982] LustreError: 203426:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6844.410110] LustreError: 203426:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6849.153386] LustreError: 205103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788857509 with bad export cookie 14709342196365078298 [ 6849.156780] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6849.163396] LustreError: 205103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6849.588604] Lustre: server umount lustre-MDT0001 complete [ 6862.373644] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 6865.871824] Key type lgssc unregistered [ 6866.109299] LNet: 207252:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6866.121963] LNetError: 207252:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6867.175211] LNet: Removed LNI 192.168.202.133@tcp [ 6868.227202] Key type .llcrypt unregistered [ 6868.230629] Key type ._llcrypt unregistered [ 6894.211141] Key type ._llcrypt registered [ 6894.217268] Key type .llcrypt registered [ 6894.379692] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 6895.333686] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6895.488545] alg: No test for adler32 (adler32-zlib) [ 6896.701334] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 6896.965274] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 6898.664226] Key type lgssc registered [ 6899.997892] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6908.213707] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6908.229187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6909.788912] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6914.574475] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6920.173128] LustreError: 208662:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6920.211657] LustreError: 208662:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6922.844513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6923.293618] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6927.638849] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6935.578447] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6941.763979] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6949.423468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6949.730593] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6954.905464] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6957.337391] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6965.522531] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6983.649840] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6983.662204] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6983.668992] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6983.682206] Lustre: Skipped 1 previous similar message [ 6983.691955] Lustre: Skipped 1 previous similar message [ 6988.616117] Lustre: server umount lustre-OST0000 complete [ 6993.896279] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6993.906156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6993.909990] Lustre: Skipped 1 previous similar message [ 6993.919512] Lustre: Skipped 1 previous similar message [ 6998.830656] Lustre: server umount lustre-MDT0000 complete [ 6999.016577] LustreError: 208658:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6999.044609] LustreError: 208658:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7003.338873] LustreError: 208644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788857663 with bad export cookie 15600333647589827529 [ 7003.352450] LustreError: 208644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7003.353572] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7004.029579] Lustre: server umount lustre-MDT0001 complete [ 7017.050121] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 7020.129461] Key type lgssc unregistered [ 7020.528756] LNet: 212344:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7020.550192] LNetError: 212344:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7020.583346] LNet: Removed LNI 192.168.202.133@tcp [ 7021.584225] Key type .llcrypt unregistered [ 7021.586880] Key type ._llcrypt unregistered [ 7039.606459] Key type ._llcrypt registered [ 7039.608575] Key type .llcrypt registered [ 7039.781956] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 7041.413770] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7041.547951] alg: No test for adler32 (adler32-zlib) [ 7042.805603] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 7043.187659] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 7044.920166] Key type lgssc registered [ 7045.911802] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7055.047471] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7055.084505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7056.776159] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7062.587682] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7066.089573] LustreError: 213608:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7066.117029] LustreError: 213608:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7071.202785] LustreError: 213607:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7072.469292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7073.140269] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7078.933680] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7087.213057] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7093.643547] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7101.324634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7101.587032] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7107.333327] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7116.451578] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7126.219523] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 7147.035391] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7147.040546] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7147.088857] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7151.579647] Lustre: server umount lustre-OST0000 complete [ 7157.217182] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7157.245487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7157.253320] Lustre: Skipped 2 previous similar messages [ 7157.272065] Lustre: Skipped 2 previous similar messages [ 7162.339032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7162.346854] Lustre: Skipped 1 previous similar message [ 7162.803196] Lustre: server umount lustre-MDT0000 complete [ 7167.458056] LustreError: 214336:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7167.481158] LustreError: 214336:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7167.845914] LustreError: 213587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788857828 with bad export cookie 1980541071053231993 [ 7167.850347] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7167.855951] LustreError: 213587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7168.327898] Lustre: server umount lustre-MDT0001 complete [ 7181.379901] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 7184.975889] Key type lgssc unregistered [ 7185.335988] LNet: 217313:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7185.351603] LNetError: 217313:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7186.414335] LNet: Removed LNI 192.168.202.133@tcp [ 7187.433551] Key type .llcrypt unregistered [ 7187.436617] Key type ._llcrypt unregistered [ 7210.958833] Key type ._llcrypt registered [ 7210.961210] Key type .llcrypt registered [ 7211.131123] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 7212.468138] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7212.619578] alg: No test for adler32 (adler32-zlib) [ 7213.770278] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 7214.171774] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 7215.888319] Key type lgssc registered [ 7217.578740] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7227.463434] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7227.478200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7229.132524] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7235.727469] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7239.652461] LustreError: 218578:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7239.686754] LustreError: 218578:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7244.770413] LustreError: 218579:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7247.085232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7247.656481] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7252.667658] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7261.720458] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7269.328816] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7278.510276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7279.175953] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7287.519658] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7296.320607] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7299.719537] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 7309.044796] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7309.109040] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7309.327933] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7316.258562] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7324.971822] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7326.770163] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7338.558760] Lustre: Permanently deactivating lustre-OST0001 [ 7338.586223] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7345.729945] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 7353.696380] Lustre: Permanently reactivating lustre-OST0001 [ 7353.725253] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7353.732834] Lustre: Skipped 2 previous similar messages [ 7361.100302] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7361.125218] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7361.137059] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7361.150504] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7361.214773] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7364.587727] Lustre: server umount lustre-OST0001 complete [ 7365.097266] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7365.102064] Lustre: Skipped 1 previous similar message [ 7365.111171] LustreError: 220402:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7365.124222] LustreError: 220402:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7369.698987] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7369.719220] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7369.739373] Lustre: Skipped 1 previous similar message [ 7369.742852] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7370.215968] LustreError: 220402:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7370.229565] LustreError: 220402:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7375.339133] LustreError: 220401:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7375.360704] LustreError: 220401:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7375.515959] Lustre: server umount lustre-OST0000 complete [ 7380.450467] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7380.463291] Lustre: Skipped 1 previous similar message [ 7380.471629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7380.485643] Lustre: Skipped 3 previous similar messages [ 7385.571523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7385.584348] Lustre: Skipped 1 previous similar message [ 7385.909696] Lustre: server umount lustre-MDT0000 complete [ 7390.537841] LustreError: 218559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788858050 with bad export cookie 14738991018205936810 [ 7390.546276] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7390.557824] LustreError: 218559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7391.088778] Lustre: server umount lustre-MDT0001 complete [ 7406.077689] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 7410.000911] Key type lgssc unregistered [ 7410.413926] LNet: 223639:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7410.432189] LNetError: 223639:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7410.455872] LNet: Removed LNI 192.168.202.133@tcp [ 7411.543542] Key type .llcrypt unregistered [ 7411.546121] Key type ._llcrypt unregistered [ 7462.524296] Key type ._llcrypt registered [ 7462.527970] Key type .llcrypt registered [ 7462.693660] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 7463.700860] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7463.770431] alg: No test for adler32 (adler32-zlib) [ 7465.338784] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 7465.699908] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 7467.456858] Key type lgssc registered [ 7469.217734] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7478.986981] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7478.999127] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7480.289601] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7480.312984] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7480.488779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7485.990776] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7495.046488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7495.268084] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7495.329186] Lustre: 225842:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7495.698845] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7503.261666] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7514.790242] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7521.732690] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7529.781383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7530.042238] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7530.442309] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7538.213910] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7538.874024] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 7546.909181] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7550.810126] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7556.392840] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7563.406608] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7573.604169] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7582.580697] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7601.286536] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7601.295292] Lustre: Skipped 2 previous similar messages [ 7636.733258] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7636.752718] Lustre: Skipped 3 previous similar messages [ 7704.480030] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7704.490688] Lustre: Skipped 7 previous similar messages [ 7733.217336] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7733.223055] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7733.238023] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7733.735764] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7733.747841] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7733.769615] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7738.854403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7738.861755] Lustre: Skipped 1 previous similar message [ 7739.332768] Lustre: server umount lustre-OST0000 complete [ 7743.970421] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7743.972893] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7743.983220] Lustre: Skipped 1 previous similar message [ 7743.997214] Lustre: Skipped 1 previous similar message [ 7749.093071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7749.105578] Lustre: Skipped 1 previous similar message [ 7749.921425] Lustre: server umount lustre-MDT0000 complete [ 7753.602611] LustreError: 227703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788858413 with bad export cookie 6233133316897567823 [ 7753.608932] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7753.613067] LustreError: 227703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7753.962343] Lustre: server umount lustre-MDT0001 complete [ 7764.939467] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 7768.046996] Key type lgssc unregistered [ 7768.453786] LNet: 230650:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7768.462675] LNetError: 230650:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7768.487889] LNet: Removed LNI 192.168.202.133@tcp [ 7769.519169] Key type .llcrypt unregistered [ 7769.524433] Key type ._llcrypt unregistered [ 7798.437723] Key type ._llcrypt registered [ 7798.442912] Key type .llcrypt registered [ 7798.581671] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 7800.087848] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7800.307120] alg: No test for adler32 (adler32-zlib) [ 7801.375953] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 7801.647294] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 7803.320595] Key type lgssc registered [ 7805.168280] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7816.098566] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7816.137721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7817.629929] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7822.947806] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7826.915630] LustreError: 232059:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7826.937490] LustreError: 232059:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7832.040136] LustreError: 232058:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7833.055895] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7833.751398] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7839.339771] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7849.083339] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7855.959715] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7863.073052] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7863.339041] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7870.711386] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7879.216734] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7887.925638] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7913.442056] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7913.446772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7913.459202] Lustre: Skipped 1 previous similar message [ 7913.998651] Lustre: server umount lustre-OST0000 complete [ 7918.564203] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7918.581615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7918.604940] Lustre: Skipped 2 previous similar messages [ 7923.682243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7923.691833] Lustre: Skipped 1 previous similar message [ 7924.708484] LustreError: 232058:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7924.733162] LustreError: 232058:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7926.011693] Lustre: server umount lustre-MDT0000 complete [ 7929.829894] LustreError: 233157:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7929.853721] LustreError: 233157:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7930.175973] LustreError: 232040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788858590 with bad export cookie 3804933942287544089 [ 7930.180486] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7930.189213] LustreError: 232040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7930.825476] Lustre: server umount lustre-MDT0001 complete [ 7943.849142] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 7947.388719] Key type lgssc unregistered [ 7947.712810] LNet: 235801:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7947.722527] LNetError: 235801:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7947.747213] LNet: Removed LNI 192.168.202.133@tcp [ 7948.762603] Key type .llcrypt unregistered [ 7948.764561] Key type ._llcrypt unregistered [ 7973.738783] Key type ._llcrypt registered [ 7973.744180] Key type .llcrypt registered [ 7973.905310] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 7975.358648] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7975.394225] alg: No test for adler32 (adler32-zlib) [ 7976.531766] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 7976.766830] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 7978.504872] Key type lgssc registered [ 7979.854536] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8254.114462] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 8257.167606] Key type lgssc unregistered [ 8257.486222] LNet: 237818:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8257.495127] LNetError: 237818:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8257.519402] LNet: Removed LNI 192.168.202.133@tcp [ 8258.445380] Key type .llcrypt unregistered [ 8258.448413] Key type ._llcrypt unregistered [ 8290.026484] Key type ._llcrypt registered [ 8290.038783] Key type .llcrypt registered [ 8290.215698] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 8308.662129] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 8310.011489] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8310.091285] alg: No test for adler32 (adler32-zlib) [ 8311.435302] Lustre: Lustre: Build Version: 2.17.58_39_ga87ca5d [ 8311.756814] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 8313.501367] Key type lgssc registered [ 8314.900815] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8349.733159] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 8035 sec ========== 05:16:48 (1788859008) [ 8352.138820] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 05:16:50 (1788859010) === [ 8356.022592] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 05:16:54 (1788859014) === [ 8405.725946] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 8409.152568] Key type lgssc unregistered [ 8409.490561] LNet: 243057:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8409.494789] LNetError: 243057:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8409.513136] LNet: Removed LNI 192.168.202.133@tcp [ 8410.791660] Key type .llcrypt unregistered [ 8410.799383] Key type ._llcrypt unregistered