[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 630778852 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001015] APIC: Switch to symmetric I/O mode setup [ 0.002000] x2apic enabled [ 0.002000] Switched APIC routing to physical x2apic. [ 0.002000] kvm-guest: setup PV IPIs [ 0.002000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.002000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.002000] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.002000] pid_max: default: 32768 minimum: 301 [ 0.002000] LSM: Security Framework initializing [ 0.002000] Yama: becoming mindful. [ 0.002000] SELinux: Initializing. [ 0.002171] *** VALIDATE selinux *** [ 0.012353] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.017561] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.019137] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.020147] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.021166] *** VALIDATE tmpfs *** [ 0.022604] *** VALIDATE proc *** [ 0.023361] *** VALIDATE cgroup *** [ 0.024021] *** VALIDATE cgroup2 *** [ 0.025347] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.026201] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.027019] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.028040] Spectre V2 : User space: Vulnerable [ 0.029013] Speculative Store Bypass: Vulnerable [ 0.033047] debug: unmapping init [mem 0xffffffffa2459000-0xffffffffa2460fff] [ 0.036371] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.038074] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.039041] ... version: 2 [ 0.040023] ... bit width: 48 [ 0.041020] ... generic registers: 4 [ 0.042027] ... value mask: 0000ffffffffffff [ 0.043031] ... max period: 00007fffffffffff [ 0.044025] ... fixed-purpose events: 3 [ 0.045017] ... event mask: 000000070000000f [ 0.046390] rcu: Hierarchical SRCU implementation. [ 0.049301] smp: Bringing up secondary CPUs ... [ 0.050967] x86: Booting SMP configuration: [ 0.051066] .... node #0, CPUs: #1 #2 #3 [ 0.060187] smp: Brought up 1 node, 4 CPUs [ 0.062027] smpboot: Max logical packages: 1 [ 0.063020] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.117507] node 0 deferred pages initialised in 52ms [ 0.122241] devtmpfs: initialized [ 0.123324] x86/mm: Memory block size: 128MB [ 0.127990] gcov: version magic: 0x41383552 [ 0.129452] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.130133] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.131513] pinctrl core: initialized pinctrl subsystem [ 0.132845] [ 0.133021] ************************************************************* [ 0.134023] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.135028] ** ** [ 0.136024] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.137023] ** ** [ 0.138032] ** This means that this kernel is built to expose internal ** [ 0.139030] ** IOMMU data structures, which may compromise security on ** [ 0.140036] ** your system. ** [ 0.141032] ** ** [ 0.142356] ** If you see this message and you are not debugging the ** [ 0.143026] ** kernel, report this immediately to your vendor! ** [ 0.144023] ** ** [ 0.145024] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.146029] ************************************************************* [ 0.148065] NET: Registered protocol family 16 [ 0.149992] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.150111] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.151098] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.152698] cpuidle: using governor menu [ 0.202040] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.204647] PCI: Using configuration type 1 for base access [ 0.206259] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.219147] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.222108] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.224173] cryptd: max_cpu_qlen set to 1000 [ 0.228534] ACPI: Added _OSI(Module Device) [ 0.230026] ACPI: Added _OSI(Processor Device) [ 0.232022] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.234020] ACPI: Added _OSI(Processor Aggregator Device) [ 0.240080] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.248482] ACPI: Interpreter enabled [ 0.250057] ACPI: PM: (supports S0 S3 S4 S5) [ 0.252017] ACPI: Using IOAPIC for interrupt routing [ 0.254189] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.258662] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.270170] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.272072] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.275030] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.278108] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.282709] acpiphp: Slot [2] registered [ 0.284296] acpiphp: Slot [5] registered [ 0.286171] acpiphp: Slot [6] registered [ 0.287163] acpiphp: Slot [7] registered [ 0.288182] acpiphp: Slot [8] registered [ 0.290287] acpiphp: Slot [9] registered [ 0.291228] acpiphp: Slot [10] registered [ 0.292132] acpiphp: Slot [3] registered [ 0.293330] acpiphp: Slot [4] registered [ 0.295112] acpiphp: Slot [11] registered [ 0.296217] acpiphp: Slot [12] registered [ 0.298129] acpiphp: Slot [13] registered [ 0.300186] acpiphp: Slot [14] registered [ 0.301281] acpiphp: Slot [15] registered [ 0.303318] acpiphp: Slot [16] registered [ 0.305323] acpiphp: Slot [17] registered [ 0.307347] acpiphp: Slot [18] registered [ 0.308135] acpiphp: Slot [19] registered [ 0.310127] acpiphp: Slot [20] registered [ 0.311362] acpiphp: Slot [21] registered [ 0.313313] acpiphp: Slot [22] registered [ 0.315158] acpiphp: Slot [23] registered [ 0.317131] acpiphp: Slot [24] registered [ 0.318169] acpiphp: Slot [25] registered [ 0.320149] acpiphp: Slot [26] registered [ 0.322163] acpiphp: Slot [27] registered [ 0.324139] acpiphp: Slot [28] registered [ 0.326190] acpiphp: Slot [29] registered [ 0.327305] acpiphp: Slot [30] registered [ 0.329107] acpiphp: Slot [31] registered [ 0.330063] PCI host bridge to bus 0000:00 [ 0.331027] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.334177] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.337031] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.339030] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.342033] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.345047] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.347188] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.350444] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.354798] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.367020] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.372056] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.375028] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.378024] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.381028] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.384816] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.387893] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.391052] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.394014] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.398022] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.410027] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.416167] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.422643] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.435021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.443023] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.478024] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.495028] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.506023] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.521029] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.540025] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.554854] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.566023] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.578026] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.601029] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.614565] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.629022] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.639176] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.665025] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.681153] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.690023] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.697023] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.724023] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.745021] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.759029] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.796029] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.825019] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.837167] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.839571] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.842543] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.844461] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.847307] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.854093] iommu: Default domain type: Passthrough [ 0.856431] SCSI subsystem initialized [ 0.860116] ACPI: bus type USB registered [ 0.862174] usbcore: registered new interface driver usbfs [ 0.864312] usbcore: registered new interface driver hub [ 0.867127] usbcore: registered new device driver usb [ 0.869231] pps_core: LinuxPPS API ver. 1 registered [ 0.871016] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.874153] PTP clock support registered [ 0.876364] EDAC MC: Ver: 3.0.0 [ 0.878237] PCI: Using ACPI for IRQ routing [ 0.881881] NetLabel: Initializing [ 0.884016] NetLabel: domain hash size = 128 [ 0.885013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.887110] NetLabel: unlabeled traffic allowed by default [ 0.890274] vgaarb: loaded [ 0.894021] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.897017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.904348] clocksource: Switched to clocksource kvm-clock [ 1.036186] VFS: Disk quotas dquot_6.6.0 [ 1.037850] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.040677] *** VALIDATE ramfs *** [ 1.041811] *** VALIDATE hugetlbfs *** [ 1.043617] pnp: PnP ACPI init [ 1.045922] pnp: PnP ACPI: found 6 devices [ 1.062654] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.066321] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.068695] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.071464] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.074541] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.077400] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.080603] NET: Registered protocol family 2 [ 1.083648] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.089024] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.092764] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.098937] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.102824] TCP: Hash tables configured (established 65536 bind 65536) [ 1.106199] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.109954] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.113394] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.116541] NET: Registered protocol family 1 [ 1.119189] RPC: Registered named UNIX socket transport module. [ 1.121648] RPC: Registered udp transport module. [ 1.123882] RPC: Registered tcp transport module. [ 1.126220] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.129393] NET: Registered protocol family 44 [ 1.131699] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.134354] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.136917] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.139368] PCI: CLS 0 bytes, default 64 [ 1.142292] Unpacking initramfs... [ 3.127916] debug: unmapping init [mem 0xffff944a3cc54000-0xffff944a3ffbffff] [ 3.132595] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.134983] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.138442] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 3.693844] Initialise system trusted keyrings [ 3.695767] Key type blacklist registered [ 3.699166] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.709832] zbud: loaded [ 3.713504] *** VALIDATE nfs *** [ 3.714816] *** VALIDATE nfs4 *** [ 3.716720] pstore: using deflate compression [ 3.720835] Platform Keyring initialized [ 3.848817] NET: Registered protocol family 38 [ 3.850775] Key type asymmetric registered [ 3.852286] Asymmetric key parser 'x509' registered [ 3.854585] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.857345] io scheduler mq-deadline registered [ 3.858660] io scheduler kyber registered [ 3.860222] io scheduler bfq registered [ 3.862547] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.864950] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.867741] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.872663] ACPI: Power Button [PWRF] [ 3.878909] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.886088] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.902173] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.913455] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.932450] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.963103] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.993937] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.998715] Non-volatile memory driver v1.3 [ 4.000678] Linux agpgart interface v0.103 [ 4.033470] virtio_blk virtio1: [vda] 134712 512-byte logical blocks (69.0 MB/65.8 MiB) [ 4.035975] vda: detected capacity change from 0 to 68972544 [ 4.060440] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.062761] vdb: detected capacity change from 0 to 1073741824 [ 4.080781] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.084764] vdc: detected capacity change from 0 to 2621440000 [ 4.158604] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.165388] vdd: detected capacity change from 0 to 2621440000 [ 4.195487] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.200354] vde: detected capacity change from 0 to 4294967296 [ 4.233650] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.236915] vdf: detected capacity change from 0 to 4294967296 [ 4.270745] libphy: Fixed MDIO Bus: probed [ 4.302828] usbcore: registered new interface driver usbserial_generic [ 4.305730] usbserial: USB Serial support registered for generic [ 4.308241] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.314408] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.318618] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.332471] mousedev: PS/2 mouse device common for all mice [ 4.351698] rtc_cmos 00:05: RTC can wake from S4 [ 4.354732] rtc_cmos 00:05: registered as rtc0 [ 4.356194] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.358562] intel_pstate: CPU model not supported [ 4.362806] hid: raw HID events driver (C) Jiri Kosina [ 4.364977] usbcore: registered new interface driver usbhid [ 4.365273] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.367305] usbhid: USB HID core driver [ 4.367623] drop_monitor: Initializing network drop monitor service [ 4.367787] Initializing XFRM netlink socket [ 4.369295] NET: Registered protocol family 10 [ 4.385824] Segment Routing with IPv6 [ 4.389311] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.390214] NET: Registered protocol family 17 [ 4.396351] mpls_gso: MPLS GSO support [ 4.402154] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.406683] RAS: Correctable Errors collector initialized. [ 4.409032] AVX version of gcm_enc/dec engaged. [ 4.410236] AES CTR mode by8 optimization enabled [ 4.490290] sched_clock: Marking stable (4490036727, 0)->(5672594266, -1182557539) [ 4.492784] registered taskstats version 1 [ 4.494362] Loading compiled-in X.509 certificates [ 4.495909] zswap: loaded using pool lzo/zbud [ 4.517761] Key type big_key registered [ 4.527907] Key type encrypted registered [ 4.529343] ima: No TPM chip found, activating TPM-bypass! [ 4.530906] ima: Allocated hash algorithm: sha1 [ 4.532425] ima: No architecture policies found [ 4.533663] evm: Initialising EVM extended attributes: [ 4.534892] evm: security.selinux [ 4.535721] evm: security.ima [ 4.536420] evm: security.capability [ 4.537296] evm: HMAC attrs: 0x1 [ 4.539307] rtc_cmos 00:05: setting system clock to 2026-03-02 16:38:22 UTC (1772469502) [ 4.545740] debug: unmapping init [mem 0xffffffffa3403000-0xffffffffa35fffff] [ 4.550240] debug: unmapping init [mem 0xffffffffa2182000-0xffffffffa2458fff] [ 4.558201] Write protecting the kernel read-only data: 28672k [ 4.561707] debug: unmapping init [mem 0xffffffffa0803000-0xffffffffa09fffff] [ 4.564501] debug: unmapping init [mem 0xffffffffa1114000-0xffffffffa11fffff] [ 4.615254] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.621860] systemd[1]: Detected virtualization kvm. [ 4.623796] systemd[1]: Detected architecture x86-64. [ 4.626388] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.665287] systemd[1]: No hostname configured. [ 4.667443] systemd[1]: Set hostname to . [ 4.670451] random: systemd: uninitialized urandom read (16 bytes read) [ 4.673714] systemd[1]: Initializing machine ID from random generator. [ 4.810508] random: ln: uninitialized urandom read (6 bytes read) [ 5.156676] random: systemd: uninitialized urandom read (16 bytes read) [ 5.162484] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 5.170586] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 5.179553] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.941905] device-mapper: uevent: version 1.0.3 [ 6.944937] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 9.430476] virtio_net virtio0 ens2: renamed from eth0 Starting dracut initqueue hook... [ 9.871346] random: fast init done [ 10.711245] scsi host0: ata_piix [ 10.870481] scsi host1: ata_piix [ 10.872121] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 10.874579] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 15.062907] random: crng init done [ 15.084657] random: 7 urandom warning(s) missed due to ratelimiting [ 18.682344] dracut-initqueue[576]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ 19.906228] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 23.040147] printk: systemd: 26 output lines suppressed due to ratelimiting [ 23.418773] SELinux: Disabled at runtime. [ 23.505325] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 23.514642] systemd[1]: Detected virtualization kvm. [ 23.516809] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 24.506349] systemd[1]: initrd-switch-root.service: Succeeded. [ 24.511613] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 24.517541] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 24.521836] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 24.526228] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 24.651098] systemd[1]: Starting Journal Service... Starting Journal Service... [ 24.661964] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... [ OK ] Listening on Process Core Dump Socket. Starting Remount Root and Kernel File Systems... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Control Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on initctl Compatibility Named Pipe. Activating swap /dev/disk/by-label/SWAP... Starting Apply Kernel Variables... [ OK ] Created slice User and Session Slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Kernel Debug File System... [ 26.165699] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Kernel Socket. Mounting POSIX Message Queue File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting udev Coldplug all Devices... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 29.002808] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 30.112917] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 30.308495] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 33.116541] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 33.269682] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…only root support (10s / no limit) [** ] A start job is running for Configur…only root support (10s / no limit) [*** ] A start job is running for Configur…only root support (11s / no limit)[ 35.866977] Key type dns_resolver registered [ *** ] A start job is running for Configur…only root support (11s / no limit)[ 36.625522] NFS: Registering the id_resolver key type [ 36.629122] Key type id_resolver registered [ 36.632805] Key type id_legacy registered [ *** ] A start job is running for Configur…only root support (12s / no limit) [ ***] A start job is running for Configur…only root support (12s / no limit) [ **] A start job is running for Configur…only root support (13s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... Starting Login Service... Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg420-server login: [ 76.412095] hrtimer: interrupt took 2245155 ns [ 104.214221] libcfs: loading out-of-tree module taints kernel. [ 104.256200] Key type ._llcrypt registered [ 104.257866] Key type .llcrypt registered [ 104.378695] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 122.831995] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 124.388575] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 124.402966] alg: No test for adler32 (adler32-zlib) [ 125.932464] Lustre: Lustre: Build Version: 2.17.50_194_gae5720b [ 126.993755] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 128.800159] Key type lgssc registered [ 130.585546] Lustre: Echo OBD driver; http://www.lustre.org/ [ 150.765206] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 196.586755] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 211.648344] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 211.718167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 213.139610] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 213.224189] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 213.334526] Lustre: lustre-MDT0000: new disk, initializing [ 213.475396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 213.509757] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 218.550043] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 233.881227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 234.003405] Lustre: 6493:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 234.062969] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 234.073683] Lustre: Skipped 1 previous similar message [ 234.239945] Lustre: lustre-MDT0001: new disk, initializing [ 234.316576] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 234.379090] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 234.401866] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 239.610135] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 245.362181] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 256.596506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 256.846904] Lustre: lustre-OST0000: new disk, initializing [ 256.851175] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 256.920665] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 259.263220] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 259.283587] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 259.408517] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 263.982942] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 278.346959] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 278.481811] Lustre: lustre-OST0001: new disk, initializing [ 278.488743] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 278.565953] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 284.621314] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 288.325684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 288.342662] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 288.401567] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 297.524534] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 303.775404] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 318.008090] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing check_logdir /tmp/testlogs/ [ 324.333180] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing yml_node [ 330.108401] Lustre: DEBUG MARKER: Client: 2.17.50.194 [ 332.660099] Lustre: DEBUG MARKER: MDS: 2.17.50.194 [ 335.855836] Lustre: DEBUG MARKER: OSS: 2.17.50.194 [ 337.972279] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Mar 2 11:43:54 EST 2026 [ 359.030559] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 360.789379] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 380.392897] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 380.401965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 380.430121] Lustre: Skipped 1 previous similar message [ 380.447364] Lustre: Skipped 3 previous similar messages [ 384.541423] Lustre: server umount lustre-MDT0000 complete [ 389.260772] LustreError: 9451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772469887 with bad export cookie 17385268611616793222 [ 389.280466] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 389.281560] LustreError: 9451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 390.629947] LustreError: 6499:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 390.630413] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 390.631334] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 390.676944] LustreError: 6499:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 390.698919] Lustre: Skipped 3 previous similar messages [ 394.657672] LustreError: 6500:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 394.658529] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 394.696354] LustreError: 6500:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 394.708870] Lustre: Skipped 2 previous similar messages [ 395.682989] Lustre: server umount lustre-MDT0001 complete [ 399.980985] Lustre: server umount lustre-OST0000 complete [ 404.498979] Lustre: server umount lustre-OST0001 complete [ 412.311651] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 423.518169] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 459.919322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 460.406243] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 460.465619] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 460.594209] Lustre: lustre-MDT0000: new disk, initializing [ 460.687329] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 460.707078] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 465.390845] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 477.098446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 477.222702] Lustre: 14971:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 477.291961] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 477.296715] Lustre: Skipped 1 previous similar message [ 477.393524] Lustre: lustre-MDT0001: new disk, initializing [ 477.507132] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 477.589205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 477.604045] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 483.312311] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 491.853473] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 495.689760] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 502.962972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 503.345515] Lustre: lustre-OST0000: new disk, initializing [ 503.349623] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 503.449211] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 505.319439] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 505.336710] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 505.510852] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 512.240531] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 521.799517] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 526.918762] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 527.160287] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 532.033718] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 532.341600] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 536.039185] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 536.044541] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 536.048356] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 536.050077] Lustre: Skipped 1 previous similar message [ 536.061795] Lustre: Skipped 1 previous similar message [ 540.771539] Lustre: server umount lustre-OST0000 complete [ 546.276481] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 546.279923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 546.295517] Lustre: Skipped 1 previous similar message [ 546.304337] Lustre: Skipped 1 previous similar message [ 551.700871] Lustre: server umount lustre-MDT0000 complete [ 556.304204] LustreError: 15760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772470054 with bad export cookie 17385268611616794755 [ 556.310842] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 556.315544] LustreError: 15760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 556.841363] Lustre: server umount lustre-MDT0001 complete [ 560.612958] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 11:47:37 (1772470057) [ 566.573958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 567.439485] LustreError: 18831:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 567.511961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 572.448707] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 572.901466] LustreError: 18832:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 578.024621] LustreError: 18831:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 583.973038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 589.897545] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 595.309960] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 599.032946] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 604.508630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 610.760138] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 617.454614] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 625.637846] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 625.650480] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 625.669672] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 625.680358] Lustre: Skipped 2 previous similar messages [ 628.716501] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 631.611690] Lustre: server umount lustre-OST0000 complete [ 635.875094] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 635.884211] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 635.893158] Lustre: Skipped 1 previous similar message [ 641.570523] Lustre: server umount lustre-MDT0000 complete [ 644.066624] LustreError: 20987:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 644.090034] LustreError: 20987:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 646.576653] LustreError: 19554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772470144 with bad export cookie 17385268611616795637 [ 646.593306] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 646.595946] LustreError: 19554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 647.061818] Lustre: server umount lustre-MDT0001 complete [ 660.740508] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 663.839454] Key type lgssc unregistered [ 664.215557] LNet: 22013:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 664.221273] LNetError: 22013:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 664.243787] LNet: Removed LNI 192.168.204.120@tcp [ 665.151173] Key type .llcrypt unregistered [ 665.152642] Key type ._llcrypt unregistered [ 695.014040] Key type ._llcrypt registered [ 695.015169] Key type .llcrypt registered [ 695.153260] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 696.561686] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 696.622460] alg: No test for adler32 (adler32-zlib) [ 697.848326] Lustre: Lustre: Build Version: 2.17.50_194_gae5720b [ 698.046770] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 699.688331] Key type lgssc registered [ 701.276938] Lustre: Echo OBD driver; http://www.lustre.org/ [ 710.865814] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 710.886814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 712.636137] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 717.694626] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 722.921496] LustreError: 23412:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 722.950563] LustreError: 23412:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 727.502078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 727.762113] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 733.255287] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 739.454980] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 744.159477] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 751.789426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 752.196766] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 757.319802] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 758.961173] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 764.824705] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 782.819641] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 782.825309] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 782.828342] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 782.841719] Lustre: Skipped 1 previous similar message [ 782.855475] Lustre: Skipped 1 previous similar message [ 787.943730] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 787.952315] Lustre: Skipped 1 previous similar message [ 788.290807] Lustre: server umount lustre-OST0000 complete [ 793.060980] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 793.095392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 793.105374] Lustre: Skipped 1 previous similar message [ 798.184564] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 798.193818] Lustre: Skipped 1 previous similar message [ 803.302306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 803.312302] Lustre: Skipped 1 previous similar message [ 806.368143] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 806.586492] Lustre: server umount lustre-MDT0000 complete [ 808.417038] LustreError: 23412:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 808.444208] LustreError: 23412:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 810.142769] LustreError: 23392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772470308 with bad export cookie 5811203537070771960 [ 810.166724] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 810.168955] LustreError: 23392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 810.642174] Lustre: server umount lustre-MDT0001 complete [ 824.700473] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 828.456746] Key type lgssc unregistered [ 828.862792] LNet: 27037:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 828.870676] LNetError: 27037:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 828.894554] LNet: Removed LNI 192.168.204.120@tcp [ 830.012625] Key type .llcrypt unregistered [ 830.015508] Key type ._llcrypt unregistered [ 857.415368] Key type ._llcrypt registered [ 857.418312] Key type .llcrypt registered [ 857.563613] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 859.379563] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 859.457594] alg: No test for adler32 (adler32-zlib) [ 860.530802] Lustre: Lustre: Build Version: 2.17.50_194_gae5720b [ 860.929976] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 862.640163] Key type lgssc registered [ 864.116153] Lustre: Echo OBD driver; http://www.lustre.org/ [ 872.737838] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 872.770807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 874.447399] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 879.422513] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 884.705747] LustreError: 28436:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 884.732609] LustreError: 28436:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 888.039946] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 888.398596] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 894.038745] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 900.573452] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 905.449533] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 921.146216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 921.467670] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 929.351592] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 935.482457] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 945.003057] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 954.348922] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 954.379202] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 954.421414] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 955.366257] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 955.369447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 955.396928] Lustre: Skipped 1 previous similar message [ 958.768053] Lustre: server umount lustre-OST0000 complete [ 965.089923] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 965.102052] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 965.132577] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 970.723605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 970.730787] Lustre: Skipped 2 previous similar messages [ 975.843827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 975.862867] Lustre: Skipped 1 previous similar message [ 977.376892] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 977.582305] Lustre: server umount lustre-MDT0000 complete [ 980.964427] LustreError: 28431:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 980.993963] LustreError: 28431:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 981.612278] LustreError: 28417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772470479 with bad export cookie 1212726577816954601 [ 981.623166] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 981.629714] LustreError: 28417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 981.976803] Lustre: server umount lustre-MDT0001 complete [ 993.723588] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 996.353995] Key type lgssc unregistered [ 996.638196] LNet: 31987:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 996.642150] LNetError: 31987:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 996.658229] LNet: Removed LNI 192.168.204.120@tcp [ 997.417410] Key type .llcrypt unregistered [ 997.420596] Key type ._llcrypt unregistered [ 1018.310594] Key type ._llcrypt registered [ 1018.313750] Key type .llcrypt registered [ 1018.417964] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 1019.351513] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1019.587305] alg: No test for adler32 (adler32-zlib) [ 1020.666632] Lustre: Lustre: Build Version: 2.17.50_194_gae5720b [ 1020.876523] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 1022.561087] Key type lgssc registered [ 1023.566967] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1030.270964] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1030.296497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1031.719299] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1035.814987] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1041.894191] LustreError: 33364:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1041.908046] LustreError: 33364:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1045.093792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1045.581841] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1050.912768] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1056.691393] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1059.632436] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1066.620978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1067.020704] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1072.976871] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1076.339529] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1079.824627] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1092.066889] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1092.076614] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1092.092676] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1097.186818] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1097.208639] Lustre: Skipped 2 previous similar messages [ 1097.614761] Lustre: server umount lustre-OST0000 complete [ 1101.799131] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1101.805621] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1101.829959] Lustre: Skipped 1 previous similar message [ 1101.846694] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1107.425663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1107.442938] Lustre: Skipped 3 previous similar messages [ 1107.951654] Lustre: server umount lustre-MDT0000 complete [ 1111.152712] LustreError: 33345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772470609 with bad export cookie 15933039265467937045 [ 1111.155421] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1111.159867] LustreError: 33345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1111.538737] Lustre: server umount lustre-MDT0001 complete [ 1122.040615] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 1125.499680] Key type lgssc unregistered [ 1125.742864] LNet: 36547:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1125.748474] LNetError: 36547:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1125.764635] LNet: Removed LNI 192.168.204.120@tcp [ 1126.536773] Key type .llcrypt unregistered [ 1126.548725] Key type ._llcrypt unregistered [ 1149.877504] Key type ._llcrypt registered [ 1149.880254] Key type .llcrypt registered [ 1149.988428] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 1151.328617] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1151.508326] alg: No test for adler32 (adler32-zlib) [ 1152.574594] Lustre: Lustre: Build Version: 2.17.50_194_gae5720b [ 1152.749737] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 1154.480209] Key type lgssc registered [ 1155.632506] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1164.425985] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1164.459360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1166.118048] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1171.522752] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1176.560523] LustreError: 37943:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1176.586737] LustreError: 37943:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1181.485633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1181.665960] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1181.919596] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1187.314953] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1192.546529] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1196.753352] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1204.152596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1204.572632] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1210.838469] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1211.791700] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1216.511813] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1223.552120] Lustre: lustre-OST0000: Not available for connect from 192.168.204.20@tcp (stopping) [ 1227.234141] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1227.239690] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1227.246603] LustreError: Skipped 1 previous similar message [ 1227.251631] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1227.296767] Lustre: Skipped 1 previous similar message [ 1228.656966] Lustre: server umount lustre-OST0000 complete [ 1252.840109] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1252.855853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1252.860647] Lustre: Skipped 2 previous similar messages [ 1256.547788] Lustre: server umount lustre-MDT0000 complete [ 1257.964746] LustreError: 37943:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1261.224167] LustreError: 37923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772470759 with bad export cookie 4859835816633694057 [ 1261.233412] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1261.238200] LustreError: 37923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1267.840777] Lustre: server umount lustre-MDT0001 complete [ 1279.980954] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 1283.742210] Key type lgssc unregistered [ 1284.141598] LNet: 41300:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1284.155297] LNetError: 41300:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1284.196785] LNet: Removed LNI 192.168.204.120@tcp [ 1285.295053] Key type .llcrypt unregistered [ 1285.303857] Key type ._llcrypt unregistered [ 1311.954696] Key type ._llcrypt registered [ 1311.959579] Key type .llcrypt registered [ 1312.071811] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 1313.555454] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1313.610194] alg: No test for adler32 (adler32-zlib) [ 1314.792823] Lustre: Lustre: Build Version: 2.17.50_194_gae5720b [ 1315.138322] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 1316.848238] Key type lgssc registered [ 1318.244921] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1328.935546] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1328.948719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1330.378152] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1335.094525] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1340.907201] LustreError: 42709:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1340.934613] LustreError: 42709:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1345.594144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1346.103280] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1350.496955] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1356.540991] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1360.892731] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1368.467620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1368.868347] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1375.482402] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1376.059740] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1381.114495] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1388.960479] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.20@tcp (stopping) [ 1389.875538] Lustre: server umount lustre-MDT0000 complete [ 1391.592291] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1391.599071] LustreError: 43421:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1391.610155] Lustre: Skipped 1 previous similar message [ 1391.655857] LustreError: 43421:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1393.572892] LustreError: 42691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772470891 with bad export cookie 14934003506933053820 [ 1393.574766] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1393.589895] LustreError: 42691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1394.032914] LustreError: 43820:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1394.084195] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.20@tcp (stopping) [ 1396.705712] LustreError: 42706:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1396.707603] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1396.752475] Lustre: Skipped 1 previous similar message [ 1396.756036] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1399.159803] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.20@tcp (stopping) [ 1399.173098] Lustre: Skipped 1 previous similar message [ 1399.994261] Lustre: server umount lustre-MDT0001 complete [ 1421.964783] Lustre: server umount lustre-OST0000 complete [ 1441.859286] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 1445.749591] Key type lgssc unregistered [ 1446.218098] LNet: 46235:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1446.224195] LNetError: 46235:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1446.248505] LNet: Removed LNI 192.168.204.120@tcp [ 1447.387633] Key type .llcrypt unregistered [ 1447.401193] Key type ._llcrypt unregistered [ 1474.009798] Key type ._llcrypt registered [ 1474.014792] Key type .llcrypt registered [ 1474.156353] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 1475.230707] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1475.274962] alg: No test for adler32 (adler32-zlib) [ 1476.604699] Lustre: Lustre: Build Version: 2.17.50_194_gae5720b [ 1476.991436] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 1478.696341] Key type lgssc registered [ 1480.200906] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1489.185309] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1489.202844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1504.800878] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1509.920243] LustreError: 47606:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.120@tcp: failed processing log, type 4: rc = -110 [ 1536.615717] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1543.103235] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1548.697216] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1716.666955] Lustre: server umount lustre-OST0000 complete [ 1733.754612] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 1736.038508] Key type lgssc unregistered [ 1736.344123] LNet: 49152:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1736.357212] LNetError: 49152:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1737.385615] LNet: Removed LNI 192.168.204.120@tcp [ 1738.073996] Key type .llcrypt unregistered [ 1738.077210] Key type ._llcrypt unregistered [ 1759.432740] Key type ._llcrypt registered [ 1759.436556] Key type .llcrypt registered [ 1759.567859] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 1760.576510] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1760.784972] alg: No test for adler32 (adler32-zlib) [ 1761.818075] Lustre: Lustre: Build Version: 2.17.50_194_gae5720b [ 1762.009654] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 1763.648227] Key type lgssc registered [ 1764.564580] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1771.224780] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1771.252854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1772.607052] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1776.253166] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1782.756536] LustreError: 50550:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1782.769444] LustreError: 50550:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1783.806738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1784.090109] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1787.865631] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1792.872452] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1796.310791] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1801.878397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1802.169298] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1807.374980] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1812.061124] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1817.133826] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1817.138077] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1817.154119] Lustre: Skipped 1 previous similar message [ 1822.193896] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1822.200844] Lustre: Skipped 1 previous similar message [ 1827.301326] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1827.306835] Lustre: Skipped 1 previous similar message [ 1828.320922] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1828.574337] Lustre: server umount lustre-OST0000 complete [ 1832.418115] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1832.430368] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1832.433841] Lustre: Skipped 1 previous similar message [ 1837.538721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1837.555180] Lustre: Skipped 2 previous similar messages [ 1838.468853] Lustre: server umount lustre-MDT0000 complete [ 1842.435884] LustreError: 51266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772471340 with bad export cookie 16403982453715737504 [ 1842.448025] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1842.457248] LustreError: 51266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1842.756583] Lustre: server umount lustre-MDT0001 complete [ 1852.854306] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 1855.544732] Key type lgssc unregistered [ 1855.812499] LNet: 53729:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1855.822351] LNetError: 53729:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1856.886174] LNet: Removed LNI 192.168.204.120@tcp [ 1857.709679] Key type .llcrypt unregistered [ 1857.711551] Key type ._llcrypt unregistered [ 1882.780901] Key type ._llcrypt registered [ 1882.783811] Key type .llcrypt registered [ 1882.910385] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 1884.307713] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1884.648956] alg: No test for adler32 (adler32-zlib) [ 1886.090273] Lustre: Lustre: Build Version: 2.17.50_194_gae5720b [ 1886.632157] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 1888.384297] Key type lgssc registered [ 1889.601420] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1897.984679] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1897.999328] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1913.568375] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1918.688233] LustreError: 55098:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.120@tcp: failed processing log, type 4: rc = -110 [ 1945.313479] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1951.775384] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1957.050348] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1963.942665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1964.583670] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1964.704948] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1969.628743] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1969.634743] LustreError: 55123:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1969.671347] LustreError: 55123:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1974.758327] LustreError: 55124:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1978.263089] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1978.691177] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1983.324274] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1989.414358] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1993.379749] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1996.261553] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1996.283872] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2001.382180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2001.391065] Lustre: Skipped 2 previous similar messages [ 2001.642368] Lustre: server umount lustre-OST0000 complete [ 2025.489974] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2025.507797] Lustre: Skipped 2 previous similar messages [ 2025.511365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2031.590140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2031.592229] Lustre: Skipped 3 previous similar messages [ 2031.894841] Lustre: server umount lustre-MDT0000 complete [ 2036.120458] LustreError: 55106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772471534 with bad export cookie 18424124456874696422 [ 2036.128587] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2036.131742] LustreError: 55106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2042.491097] Lustre: server umount lustre-MDT0001 complete [ 2052.984605] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 2056.365725] Key type lgssc unregistered [ 2056.639316] LNet: 58493:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2056.656168] LNetError: 58493:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2056.692347] LNet: Removed LNI 192.168.204.120@tcp [ 2057.594950] Key type .llcrypt unregistered [ 2057.600410] Key type ._llcrypt unregistered [ 2081.849385] Key type ._llcrypt registered [ 2081.855118] Key type .llcrypt registered [ 2081.945917] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 2083.339980] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2083.501889] alg: No test for adler32 (adler32-zlib) [ 2084.637556] Lustre: Lustre: Build Version: 2.17.50_194_gae5720b [ 2084.847495] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 2086.520354] Key type lgssc registered [ 2087.747090] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2096.078502] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2096.108184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2097.669331] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2102.610570] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2107.878917] LustreError: 59890:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2107.904210] LustreError: 59890:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2111.161990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2111.628512] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2116.843934] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2122.363409] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2126.960627] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2133.924037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2134.294369] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2141.214946] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2142.538551] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2147.844368] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2167.268358] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2167.280136] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2167.297021] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2168.302619] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2168.316117] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2168.326174] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2171.167187] Lustre: server umount lustre-OST0000 complete [ 2178.017477] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2178.022716] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2178.031624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2178.040630] Lustre: Skipped 1 previous similar message [ 2181.508585] Lustre: server umount lustre-MDT0000 complete [ 2183.656399] LustreError: 59909:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2183.674317] LustreError: 59909:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2185.137829] LustreError: 59870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772471683 with bad export cookie 5149681923052594585 [ 2185.154971] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2185.158519] LustreError: 59870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2185.513868] Lustre: server umount lustre-MDT0001 complete [ 2196.227558] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 2198.871971] Key type lgssc unregistered [ 2199.171865] LNet: 63090:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2199.177664] LNetError: 63090:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2199.193699] LNet: Removed LNI 192.168.204.120@tcp [ 2199.979493] Key type .llcrypt unregistered [ 2199.980980] Key type ._llcrypt unregistered [ 2225.593972] Key type ._llcrypt registered [ 2225.596049] Key type .llcrypt registered [ 2225.720342] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 2227.029120] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2227.085315] alg: No test for adler32 (adler32-zlib) [ 2228.258290] Lustre: Lustre: Build Version: 2.17.50_194_gae5720b [ 2228.570411] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 2230.272166] Key type lgssc registered [ 2231.401733] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2240.595611] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 12:15:37 (1772471737) [ 2247.536111] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2247.560746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2249.027026] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2253.644919] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2258.403839] LustreError: 64780:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2258.426676] LustreError: 64780:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2262.465909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2262.873515] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2267.499754] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2274.292387] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2278.584422] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2285.393176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2285.937406] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2292.514805] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2293.149883] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2299.984596] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2307.013064] Lustre: Failing over lustre-MDT0000 [ 2307.301379] Lustre: server umount lustre-MDT0000 complete [ 2308.002201] LustreError: 64774:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2308.578963] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2308.584290] Lustre: Skipped 2 previous similar messages [ 2313.076163] LustreError: 64774:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2313.102300] LustreError: 64774:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2315.685590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2315.780703] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2315.900715] Lustre: *** cfs_fail_loc=135, val=0*** [ 2315.978865] LustreError: 67529:0:(obd_config.c:880:class_setup()) setup lustre-MDT0000 failed (-2) [ 2315.987424] LustreError: 67529:0:(obd_config.c:2136:class_config_llog_handler()) MGC192.168.204.120@tcp: cfg command failed: rc = -2 [ 2315.994364] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2315.994364] [ 2316.004157] LustreError: MGC192.168.204.120@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2316.019387] LustreError: 67517:0:(tgt_mount.c:1822:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2316.031569] LustreError: 67517:0:(tgt_mount.c:2579:server_fill_super()) Unable to start targets: -2 [ 2316.039813] LustreError: 67517:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-MDT0000 [ 2316.046478] LustreError: 67517:0:(tgt_mount.c:117:server_deregister_mount()) lustre-MDT0000 not registered [ 2316.071751] Lustre: server umount lustre-MDT0000 complete [ 2316.074857] LustreError: 67517:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2318.196732] LustreError: 64776:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2318.219697] LustreError: 64776:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2321.759688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2321.873625] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2322.192398] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2322.248995] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2323.313446] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2326.008966] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2327.545074] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2327.595874] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2327.644403] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2334.179659] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2334.190552] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2334.210509] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2337.766186] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2337.766790] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2337.791560] Lustre: Skipped 1 previous similar message [ 2340.005283] Lustre: server umount lustre-OST0000 complete [ 2348.001162] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2348.007549] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2348.013419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2348.031937] Lustre: Skipped 1 previous similar message [ 2350.599449] Lustre: server umount lustre-MDT0000 complete [ 2351.009440] LustreError: 64776:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2351.034427] LustreError: 64776:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2354.897449] LustreError: 64759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772471852 with bad export cookie 9020939863407863155 [ 2354.913840] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2354.927817] LustreError: 64759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2355.307992] Lustre: server umount lustre-MDT0001 complete [ 2366.624640] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 2369.110989] Key type lgssc unregistered [ 2369.384948] LNet: 69278:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2369.390258] LNetError: 69278:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2370.419219] LNet: Removed LNI 192.168.204.120@tcp [ 2371.355363] Key type .llcrypt unregistered [ 2371.357330] Key type ._llcrypt unregistered [ 2394.797809] Key type ._llcrypt registered [ 2394.799815] Key type .llcrypt registered [ 2394.920432] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 2396.459447] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2396.521060] alg: No test for adler32 (adler32-zlib) [ 2397.736636] Lustre: Lustre: Build Version: 2.17.50_194_gae5720b [ 2398.250977] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 2400.032168] Key type lgssc registered [ 2401.240550] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2410.428041] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2410.447434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2412.034805] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2417.097125] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2421.225301] LustreError: 70678:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2421.249423] LustreError: 70678:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2426.304645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2426.339341] LustreError: 70677:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2426.881558] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2431.974834] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2438.170688] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2442.719546] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2450.880351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2451.350914] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2456.510526] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2459.162826] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2464.886910] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2471.101583] Lustre: Failing over lustre-MDT0000 [ 2471.459484] Lustre: server umount lustre-MDT0000 complete [ 2471.912133] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2471.919700] LustreError: 71884:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2471.940178] Lustre: Skipped 1 previous similar message [ 2471.986558] LustreError: 71884:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2477.031087] LustreError: 70677:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2477.075212] LustreError: 70677:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2482.165804] LustreError: 71884:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2482.178613] LustreError: 71884:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2482.583944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2482.817741] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2482.982337] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2483.055572] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.20@tcp (not set up) [ 2483.169754] LustreError: 73429:0:(obd_config.c:880:class_setup()) setup lustre-MDT0000 failed (-9) [ 2483.175768] LustreError: 73429:0:(obd_config.c:2136:class_config_llog_handler()) MGC192.168.204.120@tcp: cfg command failed: rc = -9 [ 2483.190596] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2483.190596] [ 2483.202132] LustreError: MGC192.168.204.120@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2483.227877] LustreError: 73416:0:(tgt_mount.c:1822:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2483.242015] LustreError: 73416:0:(tgt_mount.c:2579:server_fill_super()) Unable to start targets: -9 [ 2483.259122] LustreError: 73416:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-MDT0000 [ 2483.279055] LustreError: 73416:0:(tgt_mount.c:117:server_deregister_mount()) lustre-MDT0000 not registered [ 2483.319243] Lustre: server umount lustre-MDT0000 complete [ 2483.328876] LustreError: 73416:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2491.617250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2491.883381] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2492.250027] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2492.321557] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2493.301679] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2496.392258] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2497.532246] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2497.573069] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2497.608866] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2507.750493] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2507.751617] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2507.755047] Lustre: Skipped 2 previous similar messages [ 2507.763085] Lustre: Skipped 1 previous similar message [ 2510.729866] Lustre: server umount lustre-OST0000 complete [ 2517.990756] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2517.992355] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2518.003807] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2518.019151] Lustre: Skipped 1 previous similar message [ 2520.428879] Lustre: server umount lustre-MDT0000 complete [ 2523.107480] LustreError: 70678:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2523.119322] LustreError: 70678:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2524.160617] LustreError: 70658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772472022 with bad export cookie 10884960101290553085 [ 2524.164108] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2524.169591] LustreError: 70658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2524.553444] Lustre: server umount lustre-MDT0001 complete [ 2535.579177] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 2538.824262] Key type lgssc unregistered [ 2539.089874] LNet: 75179:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2539.098357] LNetError: 75179:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2539.110984] LNet: Removed LNI 192.168.204.120@tcp [ 2540.229346] Key type .llcrypt unregistered [ 2540.231913] Key type ._llcrypt unregistered [ 2554.767258] Key type ._llcrypt registered [ 2554.770797] Key type .llcrypt registered [ 2554.949277] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 2555.237284] Key type .llcrypt unregistered [ 2555.239249] Key type ._llcrypt unregistered [ 2582.147846] Key type ._llcrypt registered [ 2582.155093] Key type .llcrypt registered [ 2582.331620] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 2583.488443] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2583.517092] alg: No test for adler32 (adler32-zlib) [ 2584.913838] Lustre: Lustre: Build Version: 2.17.50_194_gae5720b [ 2585.171608] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 2586.850225] Key type lgssc registered [ 2588.301567] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2597.744511] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2597.792590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2599.401787] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2604.399762] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2609.634451] LustreError: 77049:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2609.689076] LustreError: 77049:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2614.108985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2614.481794] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2619.397363] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2627.189906] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2632.645269] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2640.000365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2640.376867] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2645.572355] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2646.802488] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2652.972716] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2665.954024] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2665.974831] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2665.984172] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2666.001686] Lustre: Skipped 1 previous similar message [ 2666.019418] Lustre: Skipped 1 previous similar message [ 2671.082323] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2671.088827] Lustre: Skipped 1 previous similar message [ 2671.555595] Lustre: server umount lustre-OST0000 complete [ 2676.198421] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2676.198434] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2676.215494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2676.220295] Lustre: Skipped 1 previous similar message [ 2676.242338] Lustre: Skipped 1 previous similar message [ 2681.317643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2681.334638] Lustre: Skipped 1 previous similar message [ 2682.052428] Lustre: server umount lustre-MDT0000 complete [ 2685.404308] LustreError: 77032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772472183 with bad export cookie 6842450931950795244 [ 2685.418351] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2685.425902] LustreError: 77032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2685.844958] Lustre: server umount lustre-MDT0001 complete [ 2698.039708] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 2701.409248] Key type lgssc unregistered [ 2701.893168] LNet: 80230:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2701.914992] LNetError: 80230:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2701.947942] LNet: Removed LNI 192.168.204.120@tcp [ 2703.071875] Key type .llcrypt unregistered [ 2703.073640] Key type ._llcrypt unregistered [ 2727.682492] Key type ._llcrypt registered [ 2727.684840] Key type .llcrypt registered [ 2727.757767] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 2728.810355] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2728.897515] alg: No test for adler32 (adler32-zlib) [ 2729.920574] Lustre: Lustre: Build Version: 2.17.50_194_gae5720b [ 2730.126640] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 2731.808387] Key type lgssc registered [ 2733.657266] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2744.495089] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2744.524657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2746.036382] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2751.132254] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2755.574788] LustreError: 81628:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2755.599302] LustreError: 81628:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2760.674452] LustreError: 81629:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2761.067604] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2761.488865] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2766.339322] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2771.948519] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2775.425449] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2782.161685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2782.611981] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2789.497909] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2790.849582] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 2795.839293] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2804.702250] Lustre: server umount lustre-OST0000 complete [ 2811.366111] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2811.375657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2811.378044] Lustre: Skipped 1 previous similar message [ 2811.394583] Lustre: Skipped 1 previous similar message [ 2816.486948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2816.502537] Lustre: Skipped 1 previous similar message [ 2818.919142] Lustre: server umount lustre-MDT0000 complete [ 2821.606575] LustreError: 81623:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2821.624293] LustreError: 81623:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2822.636585] Lustre: 80940:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772472304/real 1772472304] req@000000007679bf9e x1858571838972928/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1772472320 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2822.685841] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2823.776331] Lustre: 80938:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772472305/real 1772472305] req@0000000018ba5aee x1858571838973184/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1772472321 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2824.092450] LustreError: 81608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772472322 with bad export cookie 16087066432358841778 [ 2824.096078] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2824.102784] LustreError: 81608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2824.467929] Lustre: server umount lustre-MDT0001 complete [ 2837.042512] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 2839.875150] Key type lgssc unregistered [ 2840.243037] LNet: 84810:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2840.260313] LNetError: 84810:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2840.280259] LNet: Removed LNI 192.168.204.120@tcp [ 2841.139117] Key type .llcrypt unregistered [ 2841.140692] Key type ._llcrypt unregistered [ 2868.226703] Key type ._llcrypt registered [ 2868.229573] Key type .llcrypt registered [ 2868.352600] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 2869.384849] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2869.460672] alg: No test for adler32 (adler32-zlib) [ 2870.610758] Lustre: Lustre: Build Version: 2.17.50_194_gae5720b [ 2870.787475] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 2872.448284] Key type lgssc registered [ 2873.582959] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2880.660490] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2880.703732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2882.304202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2886.877241] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2892.769912] LustreError: 86214:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2892.800158] LustreError: 86214:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2895.776916] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2896.351586] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2901.125858] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2906.939458] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2911.136579] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2919.378992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2919.929411] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2926.252672] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2927.096922] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 2932.395943] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2947.555336] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2947.581383] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2947.608660] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2947.609731] Lustre: Skipped 1 previous similar message [ 2952.547458] Lustre: server umount lustre-OST0000 complete [ 2957.795438] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2957.799624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2957.818187] Lustre: Skipped 1 previous similar message [ 2957.837499] Lustre: Skipped 2 previous similar messages [ 2962.923719] LustreError: 86214:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2963.132372] Lustre: server umount lustre-MDT0000 complete [ 2967.357475] LustreError: 86196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772472465 with bad export cookie 12295386278762619415 [ 2967.368686] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2967.375757] LustreError: 86196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2967.787903] Lustre: server umount lustre-MDT0001 complete [ 2980.353192] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 2983.588635] Key type lgssc unregistered [ 2983.854748] LNet: 89416:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2983.860829] LNetError: 89416:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2984.936865] LNet: Removed LNI 192.168.204.120@tcp [ 2985.853378] Key type .llcrypt unregistered [ 2985.856625] Key type ._llcrypt unregistered [ 3009.980765] Key type ._llcrypt registered [ 3009.983180] Key type .llcrypt registered [ 3010.114914] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 3011.268603] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3011.315158] alg: No test for adler32 (adler32-zlib) [ 3012.356965] Lustre: Lustre: Build Version: 2.17.50_194_gae5720b [ 3012.723283] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 3014.456291] Key type lgssc registered [ 3016.056596] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3025.130068] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3025.147745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3040.737774] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3045.856215] LustreError: 90790:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.120@tcp: failed processing log, type 4: rc = -110 [ 3072.610798] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3079.367807] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3086.217742] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3092.910581] Lustre: server umount lustre-OST0000 complete [ 3108.350343] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 12:30:04 (1772472604) [ 3119.304257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3119.650846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3124.706660] LustreError: 92317:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3124.726629] LustreError: 92317:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3124.822712] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3129.829991] LustreError: 92316:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3134.949234] LustreError: 92317:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3135.682568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3136.254143] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3142.176779] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3148.777685] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3153.263894] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3160.062773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3160.456653] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3164.601897] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 3167.269038] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3174.076974] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3187.169276] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3187.175455] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3187.186361] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3190.245051] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3190.247922] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3190.271622] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3192.475480] Lustre: server umount lustre-OST0000 complete [ 3197.922178] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3197.932252] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3197.957960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3197.976598] Lustre: Skipped 1 previous similar message [ 3200.483562] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3200.489422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3200.511847] Lustre: Skipped 1 previous similar message [ 3202.697400] Lustre: server umount lustre-MDT0000 complete [ 3205.606646] LustreError: 92316:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3205.645110] LustreError: 92316:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3206.906457] LustreError: 92297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772472704 with bad export cookie 1344771184586116524 [ 3206.922603] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3207.634716] Lustre: server umount lustre-MDT0001 complete [ 3218.779643] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 3221.329506] Key type lgssc unregistered [ 3221.606123] LNet: 95666:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3221.610328] LNetError: 95666:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3221.623113] LNet: Removed LNI 192.168.204.120@tcp [ 3222.606979] Key type .llcrypt unregistered [ 3222.608521] Key type ._llcrypt unregistered [ 3247.202458] Key type ._llcrypt registered [ 3247.208206] Key type .llcrypt registered [ 3247.311872] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 3248.745991] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3248.774067] alg: No test for adler32 (adler32-zlib) [ 3249.838524] Lustre: Lustre: Build Version: 2.17.50_194_gae5720b [ 3250.148408] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 3251.824326] Key type lgssc registered [ 3253.179187] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3261.495617] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3261.507760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3263.050948] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3267.717624] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3272.177215] LustreError: 97060:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3272.185083] LustreError: 97060:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3275.614738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3276.134711] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3281.696561] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3289.549791] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3292.895197] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3299.805323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3300.439504] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3306.921752] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3307.682169] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 3312.655872] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3322.342629] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3322.350200] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3322.363885] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3323.361645] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3323.373536] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3323.374549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3323.395961] Lustre: Skipped 1 previous similar message [ 3328.482243] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3328.488551] Lustre: Skipped 1 previous similar message [ 3333.602631] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3336.160157] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3336.641899] Lustre: server umount lustre-OST0000 complete [ 3342.817363] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3342.835149] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3342.853670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3342.871060] Lustre: Skipped 1 previous similar message [ 3346.821496] Lustre: server umount lustre-MDT0000 complete [ 3348.967670] LustreError: 97060:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3348.996772] LustreError: 97060:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3351.207526] LustreError: 97042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772472849 with bad export cookie 9041187424228601989 [ 3351.213740] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3351.533518] Lustre: server umount lustre-MDT0001 complete [ 3363.533499] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 3367.513680] Key type lgssc unregistered [ 3367.905715] LNet: 100246:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3367.909158] LNetError: 100246:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3368.949301] LNet: Removed LNI 192.168.204.120@tcp [ 3370.277856] Key type .llcrypt unregistered [ 3370.282636] Key type ._llcrypt unregistered [ 3390.581834] Key type ._llcrypt registered [ 3390.584789] Key type .llcrypt registered [ 3390.675908] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 3391.931896] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3392.072361] alg: No test for adler32 (adler32-zlib) [ 3393.150825] Lustre: Lustre: Build Version: 2.17.50_194_gae5720b [ 3393.410934] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 3395.144380] Key type lgssc registered [ 3396.487945] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3406.506283] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3406.528169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3422.178453] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3427.296509] LustreError: 101475:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.120@tcp: failed processing log, type 4: rc = -110 [ 3453.985476] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3462.174882] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3469.061833] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3476.140654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3476.344777] LustreError: 100810:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000006287083c x1858572534155520/t0(0) o250->MGC192.168.204.120@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3476.634669] LustreError: MGC192.168.204.120@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3476.657461] LustreError: 102387:0:(tgt_mount.c:1822:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3476.664338] LustreError: 102387:0:(tgt_mount.c:2579:server_fill_super()) Unable to start targets: -2 [ 3476.669674] LustreError: 102387:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-MDT0000 [ 3476.774903] Lustre: server umount lustre-MDT0000 complete [ 3476.777713] LustreError: 102387:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3497.952372] Lustre: 100812:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772472979/real 1772472979] req@00000000aa384736 x1858572534156672/t0(0) o400->MGC192.168.204.120@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1772472995 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3497.994625] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3498.022459] LustreError: 101487:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3518.653275] Lustre: server umount lustre-OST0000 complete [ 3528.888288] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 3539.252339] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 3578.741421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3579.029888] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3579.069479] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3579.205597] Lustre: lustre-MDT0000: new disk, initializing [ 3579.312368] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3579.328683] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3584.453561] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3597.010725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3597.140884] Lustre: 105149:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3597.195435] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3597.199396] Lustre: Skipped 1 previous similar message [ 3597.275653] Lustre: lustre-MDT0001: new disk, initializing [ 3597.365530] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3597.396267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3597.402417] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3601.664206] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3609.920153] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3614.884637] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3623.685125] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3624.194574] Lustre: lustre-OST0000: new disk, initializing [ 3624.204598] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3624.358918] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3632.247596] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3632.273742] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3632.304521] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3632.487329] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3642.799574] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3648.692230] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3648.947207] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3654.449923] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3654.753737] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3658.211032] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3658.213260] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3658.221970] LustreError: Skipped 1 previous similar message [ 3658.246750] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3658.248285] Lustre: Skipped 1 previous similar message [ 3663.344442] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3663.361909] Lustre: Skipped 2 previous similar messages [ 3663.674182] Lustre: server umount lustre-OST0000 complete [ 3668.459971] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3668.470834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3668.485974] Lustre: Skipped 1 previous similar message [ 3668.513357] Lustre: Skipped 1 previous similar message [ 3673.571165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3673.578934] Lustre: Skipped 1 previous similar message [ 3675.166217] Lustre: server umount lustre-MDT0000 complete [ 3679.571124] LustreError: 105926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772473177 with bad export cookie 2682480253231933809 [ 3679.572817] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3679.595244] LustreError: 105926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3679.903091] Lustre: server umount lustre-MDT0001 complete [ 3689.566300] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 12:39:46 (1772473186) [ 3692.406565] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3709.129476] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 3719.047139] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 3764.568211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3764.999265] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3765.072939] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3765.240565] Lustre: lustre-MDT0000: new disk, initializing [ 3765.383187] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3765.434465] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3771.572660] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3783.267653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3783.404571] Lustre: 111613:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3783.432683] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3783.434715] Lustre: Skipped 1 previous similar message [ 3783.491634] Lustre: lustre-MDT0001: new disk, initializing [ 3783.701736] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3783.771692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3783.786035] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3790.741537] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3799.339841] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3803.867540] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3811.672961] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3811.970391] Lustre: lustre-OST0000: new disk, initializing [ 3811.974817] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3812.092751] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3819.530424] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3820.675260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3820.691167] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3826.252915] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3831.337725] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3836.395311] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3836.803494] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3842.254741] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3842.473947] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3846.626676] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3846.640606] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3846.648950] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3846.651914] Lustre: Skipped 1 previous similar message [ 3846.665488] Lustre: Skipped 1 previous similar message [ 3850.424029] Lustre: server umount lustre-OST0000 complete [ 3855.332453] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3855.351313] LustreError: Skipped 1 previous similar message [ 3855.363746] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3855.400220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3861.203256] Lustre: server umount lustre-MDT0000 complete [ 3861.992190] LustreError: 112822:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3862.020647] LustreError: 112822:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3865.297712] LustreError: 111604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772473363 with bad export cookie 2682480253231934684 [ 3865.303294] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3865.309721] LustreError: 111604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3865.626883] Lustre: server umount lustre-MDT0001 complete [ 3874.262616] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3874.634844] LustreError: 115426:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3874.707760] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3878.449926] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3879.908419] LustreError: 115427:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3885.030983] LustreError: 115426:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3887.956661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3893.706761] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3899.263312] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3903.024898] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3909.836121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3916.627694] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3922.350541] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3929.069763] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3933.673710] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3933.682982] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3933.710666] Lustre: Skipped 1 previous similar message [ 3933.725448] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3933.746670] Lustre: Skipped 2 previous similar messages [ 3935.725171] LustreError: 117018:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3935.901290] Lustre: server umount lustre-OST0000 complete [ 3948.224895] Lustre: server umount lustre-MDT0000 complete [ 3951.074317] LustreError: 115423:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3951.092545] LustreError: 115423:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3952.368356] LustreError: 115407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772473450 with bad export cookie 2682480253231935573 [ 3952.378023] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3952.391741] LustreError: 115407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3952.642790] Lustre: server umount lustre-MDT0001 complete [ 3962.239815] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 3964.925499] Key type lgssc unregistered [ 3965.136053] LNet: 118701:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3965.142703] LNetError: 118701:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3965.162277] LNet: Removed LNI 192.168.204.120@tcp [ 3965.866684] Key type .llcrypt unregistered [ 3965.869229] Key type ._llcrypt unregistered [ 3981.416265] Key type ._llcrypt registered [ 3981.418516] Key type .llcrypt registered [ 3981.515805] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 3994.405385] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 3995.537661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3995.644265] alg: No test for adler32 (adler32-zlib) [ 3996.693780] Lustre: Lustre: Build Version: 2.17.50_194_gae5720b [ 3996.914051] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 3998.568161] Key type lgssc registered [ 3999.617453] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4030.557123] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4030.592866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4031.977157] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4032.022770] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4032.117433] Lustre: lustre-MDT0000: new disk, initializing [ 4032.216448] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4032.239813] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4035.630592] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4044.992625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4045.142138] Lustre: 121820:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4045.198991] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4045.203197] Lustre: Skipped 1 previous similar message [ 4045.324358] Lustre: lustre-MDT0001: new disk, initializing [ 4045.429965] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4045.462643] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4045.475089] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4049.472136] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4055.516376] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4058.086058] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4063.069886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4063.267606] Lustre: lustre-OST0000: new disk, initializing [ 4063.271709] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4063.349216] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4067.788800] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4070.441496] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4070.444961] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4070.544308] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4074.575339] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4078.047349] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4078.185910] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4081.591989] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4081.744220] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4085.730519] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4085.736083] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4085.756180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4086.259416] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4086.266939] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4086.286734] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4086.292475] Lustre: Skipped 1 previous similar message [ 4089.627658] Lustre: server umount lustre-OST0000 complete [ 4096.481534] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4096.484445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4096.501833] Lustre: Skipped 1 previous similar message [ 4096.521265] Lustre: Skipped 1 previous similar message [ 4101.602301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4101.607987] Lustre: Skipped 1 previous similar message [ 4106.720978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4106.725346] Lustre: Skipped 1 previous similar message [ 4107.232862] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4107.390978] Lustre: server umount lustre-MDT0000 complete [ 4110.621452] LustreError: 121811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772473608 with bad export cookie 17965470063552227424 [ 4110.647903] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4110.924710] Lustre: server umount lustre-MDT0001 complete [ 4117.873340] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 12:46:54 (1772473614) [ 4123.342238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4123.798920] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4127.411332] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4129.251301] LustreError: 125819:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4129.259607] LustreError: 125819:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4133.351337] LustreError: 125820:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4134.465855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4134.750717] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4138.495226] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4142.437434] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4145.171687] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4149.220038] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4149.228757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4149.231213] Lustre: Skipped 1 previous similar message [ 4149.247116] Lustre: Skipped 1 previous similar message [ 4153.062746] Lustre: server umount lustre-MDT0000 complete [ 4154.337659] LustreError: 125814:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4154.358483] LustreError: 125814:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4155.948615] LustreError: 125801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772473653 with bad export cookie 17965470063552228299 [ 4155.953464] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4155.969795] LustreError: 125801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 4156.271939] Lustre: server umount lustre-MDT0001 complete [ 4162.553912] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 12:47:39 (1772473659) [ 4167.897712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4183.520925] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4188.642799] LustreError: 128006:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.120@tcp: failed processing log, type 4: rc = -110 [ 4214.368321] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4218.162934] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4221.901795] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4223.415410] Lustre: server umount lustre-OST0000 complete [ 4228.671960] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 12:48:45 (1772473725) [ 4232.908216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4233.157726] LustreError: 129337:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4233.204912] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4235.952934] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4238.308546] LustreError: 129338:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4241.020409] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4243.735714] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4247.390183] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4249.499645] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4253.585672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4257.523716] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4260.995214] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4271.587348] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4271.590402] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4271.595277] Lustre: Skipped 2 previous similar messages [ 4271.599183] Lustre: Skipped 1 previous similar message [ 4276.509264] Lustre: server umount lustre-MDT0000 complete [ 4276.708847] LustreError: 131488:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4276.716745] LustreError: 131488:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4278.830915] LustreError: 129318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772473776 with bad export cookie 17965470063552228964 [ 4278.843908] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4279.075703] Lustre: server umount lustre-MDT0001 complete [ 4291.897407] Lustre: server umount lustre-OST0000 complete [ 4297.118872] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 12:49:54 (1772473794) [ 4301.483482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4301.701763] LustreError: 132647:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4301.738940] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4301.743250] Lustre: Skipped 2 previous similar messages [ 4304.312291] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4309.463505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4311.966461] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4314.856489] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4317.176361] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4321.218389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4324.701996] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4326.570555] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4327.926486] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4330.588552] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4330.735354] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4333.543767] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4335.769979] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 2 sec [ 4337.121986] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4337.128974] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4337.135063] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4337.138712] Lustre: Skipped 1 previous similar message [ 4340.197593] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4343.078975] Lustre: server umount lustre-OST0000 complete [ 4345.828670] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4351.848157] Lustre: server umount lustre-MDT0000 complete [ 4354.011852] LustreError: 134265:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772473851 with bad export cookie 17965470063552230035 [ 4354.017099] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4354.024231] LustreError: 134265:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 4354.251230] Lustre: server umount lustre-MDT0001 complete [ 4359.818171] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 12:50:56 (1772473856) [ 4363.842345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4379.360610] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4384.480482] LustreError: 136453:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.120@tcp: failed processing log, type 4: rc = -110 [ 4410.080359] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4410.085969] Lustre: Skipped 2 previous similar messages [ 4413.835530] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4417.492498] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4421.654446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4421.940464] LustreError: 136478:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4421.954984] LustreError: 136478:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4422.049081] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4424.463831] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4429.416044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4432.255478] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4435.942218] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4438.386695] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4441.497764] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4441.625773] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4445.257658] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4445.393349] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4450.273511] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4450.276094] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4450.276748] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4450.276753] Lustre: Skipped 5 previous similar messages [ 4450.313315] Lustre: Skipped 5 previous similar messages [ 4452.773925] Lustre: server umount lustre-MDT0000 complete [ 4455.312748] LustreError: 136459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772473953 with bad export cookie 17965470063552231064 [ 4455.321631] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4455.333194] LustreError: 136459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4455.527123] Lustre: server umount lustre-MDT0001 complete [ 4464.005550] Lustre: server umount lustre-OST0000 complete [ 4469.049841] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 12:52:46 (1772473966) [ 4473.150487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4488.675252] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4493.794898] LustreError: 140269:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.120@tcp: failed processing log, type 4: rc = -110 [ 4523.572744] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4527.396773] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4531.789903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4532.078260] LustreError: 140294:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4532.103344] LustreError: 140294:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4532.201339] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4534.795775] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4540.746072] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4541.014208] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4541.017645] Lustre: Skipped 4 previous similar messages [ 4544.101796] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4547.745461] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4550.329652] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4554.861576] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4554.973588] Lustre: lustre-OST0001: new disk, initializing [ 4554.976173] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4559.357402] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4562.989572] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4563.001206] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4563.087846] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4564.847559] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4567.809539] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4567.942252] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4570.517832] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4570.667595] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4573.161537] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4573.163283] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4573.171339] Lustre: Skipped 2 previous similar messages [ 4578.005368] Lustre: server umount lustre-OST0000 complete [ 4583.403816] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4583.414346] Lustre: Skipped 6 previous similar messages [ 4586.776792] Lustre: server umount lustre-OST0001 complete [ 4595.411452] Lustre: server umount lustre-MDT0000 complete [ 4597.810875] LustreError: 140277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772474095 with bad export cookie 17965470063552232016 [ 4597.813298] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4597.820916] LustreError: 140277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4598.062964] Lustre: server umount lustre-MDT0001 complete [ 4614.821815] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 12:55:11 (1772474111) [ 4615.997678] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4616.996491] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4618.130986] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 12:55:15 (1772474115) [ 4622.870990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4623.156858] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4623.188586] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4626.278423] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4631.807380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4631.870710] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4631.903419] Lustre: 146212:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4635.177460] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4638.947186] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4641.645293] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4645.567760] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4645.741620] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4649.566684] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4652.114970] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4653.042670] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4655.746150] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4655.884195] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4658.708290] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4658.823084] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4662.245263] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4662.255219] LustreError: Skipped 1 previous similar message [ 4662.259808] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4662.270875] Lustre: Skipped 4 previous similar messages [ 4666.197682] Lustre: server umount lustre-OST0000 complete [ 4684.576170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4688.477478] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4691.668626] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4694.753919] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4702.124805] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4702.133225] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4702.137191] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4702.291508] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 7 sec [ 4704.969410] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4705.140373] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4707.318326] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9ce058873800.ost_server_uuid 50 [ 4708.205221] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9ce058873800.ost_server_uuid in FULL state after 0 sec [ 4708.338576] LustreError: 147132:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.204.20@tcp) returned error from blocking AST (req@00000000f9eebe5c x1858573166664064 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000d00a7070/0xf9522bd670f203aa lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.20@tcp remote: 0x5761362c08947a46 expref: 5 pid: 147132 timeout: 4808 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4708.360884] LustreError: lustre-MDT0000: A client on nid 192.168.204.20@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4708.366678] LustreError: 146207:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.204.20@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000d00a7070/0xf9522bd670f203aa lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.20@tcp remote: 0x5761362c08947a46 expref: 6 pid: 147132 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4716.334403] Lustre: server umount lustre-OST0000 complete [ 4726.910086] Lustre: server umount lustre-MDT0000 complete [ 4727.778184] LustreError: 147133:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4727.783888] LustreError: 147133:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 4728.986694] LustreError: 146202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772474226 with bad export cookie 17965470063552233199 [ 4728.987490] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4728.995580] LustreError: 146202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4735.298828] Lustre: server umount lustre-MDT0001 complete [ 4741.891532] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 4743.530959] Key type lgssc unregistered [ 4743.762573] LNet: 151735:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4743.775274] LNetError: 151735:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4743.786851] LNet: Removed LNI 192.168.204.120@tcp [ 4744.290852] Key type .llcrypt unregistered [ 4744.292361] Key type ._llcrypt unregistered [ 4758.654519] Key type ._llcrypt registered [ 4758.656937] Key type .llcrypt registered [ 4758.741298] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 4759.604750] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4759.624073] alg: No test for adler32 (adler32-zlib) [ 4760.592183] Lustre: Lustre: Build Version: 2.17.50_194_gae5720b [ 4760.741869] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 4762.352226] Key type lgssc registered [ 4763.114381] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4768.175601] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4768.191172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4769.471297] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4772.082851] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4777.061755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4777.273912] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4779.722944] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4783.547540] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4786.150979] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4790.559604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4790.790886] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4794.904381] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4794.943064] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 4799.023832] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4807.595489] Lustre: Failing over lustre-MDT0000 [ 4807.775264] Lustre: server umount lustre-MDT0000 complete [ 4808.163465] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4808.167612] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4808.174889] LustreError: 153820:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4808.181868] LustreError: 153820:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4812.258723] LustreError: 153111:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4812.267752] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4812.272112] LustreError: 153111:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4812.287334] Lustre: Skipped 1 previous similar message [ 4817.377703] LustreError: 153110:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4817.396556] LustreError: 153110:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4819.131462] LustreError: 153090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.20@tcp arrived at 1772474317 with bad export cookie 6311633175784570618 [ 4822.509150] LustreError: 153820:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4822.523414] LustreError: 153820:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4823.813817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4823.891623] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4824.078405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4824.116746] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4827.047960] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4828.622618] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4828.625116] Lustre: lustre-MDT0000: Denying connection for new client 6a01bb02-dca1-4b20-a2d3-f8370e4a0503 (at 192.168.204.20@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 4829.162046] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4833.645757] Lustre: lustre-MDT0000: Denying connection for new client 6a01bb02-dca1-4b20-a2d3-f8370e4a0503 (at 192.168.204.20@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:04 [ 4837.858136] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4837.867246] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4837.882371] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4839.399846] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4839.404430] Lustre: Skipped 1 previous similar message [ 4842.141054] Lustre: server umount lustre-OST0000 complete [ 4844.792805] LustreError: 156489:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 4844.796619] Lustre: 155853:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4844.802355] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 4844.815818] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 4844.823499] LustreError: 155853:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000a24ae6ac x1858573967596672/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4844.830349] LustreError: 155853:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 4844.835049] LustreError: 155853:0:(fid_request.c:316:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 4844.839454] Lustre: lustre-MDT0000: Recovery over after 0:16, of 2 clients 0 recovered and 2 were evicted. [ 4845.043516] Lustre: server umount lustre-MDT0000 complete [ 4847.700272] LustreError: 153090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772474345 with bad export cookie 6311633175784570723 [ 4847.704235] LustreError: 153090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4847.705338] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4847.719198] LustreError: 152419:0:(import.c:706:ptlrpc_connect_import_locked()) can't connect to a closed import [ 4853.942901] Lustre: server umount lustre-MDT0001 complete [ 4861.078858] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 4862.858713] Key type lgssc unregistered [ 4863.060673] LNet: 157225:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4863.074118] LNetError: 157225:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4863.089878] LNet: Removed LNI 192.168.204.120@tcp [ 4863.593212] Key type .llcrypt unregistered [ 4863.594360] Key type ._llcrypt unregistered [ 4878.515732] Key type ._llcrypt registered [ 4878.517746] Key type .llcrypt registered [ 4878.591699] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 4879.450937] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4879.502895] alg: No test for adler32 (adler32-zlib) [ 4880.465342] Lustre: Lustre: Build Version: 2.17.50_194_gae5720b [ 4880.618719] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 4882.272884] Key type lgssc registered [ 4883.016509] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4888.556868] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4888.567327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4889.935106] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4892.894665] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4897.796106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4898.034077] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4900.656940] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4904.045927] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4906.380123] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4910.522771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4910.776074] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4914.099150] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4914.863520] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 4917.605627] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4923.373584] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4923.374576] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4928.483368] LustreError: 160191:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4928.498335] LustreError: 160191:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4928.557856] Lustre: server umount lustre-OST0000 complete [ 4933.603712] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4933.604606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4933.609265] Lustre: Skipped 1 previous similar message [ 4933.611250] Lustre: Skipped 1 previous similar message [ 4936.803605] Lustre: server umount lustre-MDT0000 complete [ 4938.722094] LustreError: 158600:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4938.731366] LustreError: 158600:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4939.025543] LustreError: 158579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772474436 with bad export cookie 3596351221636682452 [ 4939.026513] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4939.031341] LustreError: 158579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4939.229630] Lustre: server umount lustre-MDT0001 complete [ 4944.488183] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 4946.081533] Key type lgssc unregistered [ 4946.272301] LNet: 161774:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4946.274881] LNetError: 161774:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4947.302401] LNet: Removed LNI 192.168.204.120@tcp [ 4947.778906] Key type .llcrypt unregistered [ 4947.783933] Key type ._llcrypt unregistered [ 4965.475239] Key type ._llcrypt registered [ 4965.477664] Key type .llcrypt registered [ 4965.537091] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 4966.203142] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4966.221236] alg: No test for adler32 (adler32-zlib) [ 4967.130046] Lustre: Lustre: Build Version: 2.17.50_194_gae5720b [ 4967.269605] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 4968.872216] Key type lgssc registered [ 4969.611688] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4974.203688] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4974.210823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4975.482160] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4977.648130] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4982.539118] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4982.766978] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4985.413582] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4988.919395] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4991.222326] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4995.288515] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4995.481737] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4998.730086] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5000.619963] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5001.537802] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5012.964056] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5012.968058] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5012.969120] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5012.975268] Lustre: Skipped 1 previous similar message [ 5017.761900] Lustre: server umount lustre-OST0000 complete [ 5023.203986] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5023.212237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5023.214602] Lustre: Skipped 1 previous similar message [ 5026.028579] Lustre: server umount lustre-MDT0000 complete [ 5027.978054] LustreError: 163428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772474525 with bad export cookie 6002392243891374542 [ 5027.984992] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5027.987440] LustreError: 163428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5028.143812] Lustre: server umount lustre-MDT0001 complete [ 5035.000888] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 5036.678838] Key type lgssc unregistered [ 5036.874744] LNet: 166626:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5036.880737] LNetError: 166626:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5037.927899] LNet: Removed LNI 192.168.204.120@tcp [ 5038.372413] Key type .llcrypt unregistered [ 5038.374094] Key type ._llcrypt unregistered [ 5049.754807] Key type ._llcrypt registered [ 5049.756376] Key type .llcrypt registered [ 5049.818257] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 5050.359270] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5050.367352] alg: No test for adler32 (adler32-zlib) [ 5051.238655] Lustre: Lustre: Build Version: 2.17.50_194_gae5720b [ 5051.341503] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 5052.952165] Key type lgssc registered [ 5053.448473] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5058.556556] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5058.565189] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5059.754720] Lustre: *** cfs_fail_loc=135, val=0*** [ 5059.804227] LustreError: 168110:0:(obd_config.c:880:class_setup()) setup lustre-MDT0000 failed (-2) [ 5059.807806] LustreError: 168110:0:(obd_config.c:2136:class_config_llog_handler()) MGC192.168.204.120@tcp: cfg command failed: rc = -2 [ 5059.812713] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5059.812713] [ 5059.817201] LustreError: MGC192.168.204.120@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5059.824783] LustreError: 168071:0:(tgt_mount.c:1822:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5059.829194] LustreError: 168071:0:(tgt_mount.c:2579:server_fill_super()) Unable to start targets: -2 [ 5059.833822] LustreError: 168071:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-MDT0000 [ 5059.837245] LustreError: 168071:0:(tgt_mount.c:117:server_deregister_mount()) lustre-MDT0000 not registered [ 5059.872855] Lustre: server umount lustre-MDT0000 complete [ 5059.874522] LustreError: 168071:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5062.912158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5063.157841] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5064.869342] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5068.541603] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5068.710116] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5070.242095] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5072.278446] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5073.890280] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5078.946458] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5078.947369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5078.952588] Lustre: Skipped 1 previous similar message [ 5078.956829] Lustre: Skipped 1 previous similar message [ 5084.129687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5084.132740] Lustre: Skipped 1 previous similar message [ 5089.248123] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5089.249542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5089.254420] Lustre: Skipped 1 previous similar message [ 5089.335883] Lustre: server umount lustre-MDT0000 complete [ 5091.263133] LustreError: 168461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772474589 with bad export cookie 8934797492783903887 [ 5091.266449] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5091.273487] LustreError: 168461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5091.453184] Lustre: server umount lustre-MDT0001 complete [ 5096.289070] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 5097.668125] Key type lgssc unregistered [ 5097.842640] LNet: 170558:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5097.847443] LNetError: 170558:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5097.858084] LNet: Removed LNI 192.168.204.120@tcp [ 5098.295246] Key type .llcrypt unregistered [ 5098.296536] Key type ._llcrypt unregistered [ 5107.864498] Key type ._llcrypt registered [ 5107.866249] Key type .llcrypt registered [ 5107.926838] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 5108.068035] Key type .llcrypt unregistered [ 5108.070291] Key type ._llcrypt unregistered [ 5117.472165] Key type ._llcrypt registered [ 5117.473926] Key type .llcrypt registered [ 5117.524128] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 5117.955477] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5117.970520] alg: No test for adler32 (adler32-zlib) [ 5118.906485] Lustre: Lustre: Build Version: 2.17.50_194_gae5720b [ 5119.063930] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 5120.688163] Key type lgssc registered [ 5121.377096] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5125.979822] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5125.996798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5141.472405] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5146.592317] LustreError: 172631:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.120@tcp: failed processing log, type 4: rc = -110 [ 5173.280271] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5176.573764] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5179.858456] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5183.024753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5183.264489] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5183.297776] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5184.936482] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5189.020099] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5189.196204] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5191.348510] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5194.230333] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5195.771361] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5196.942433] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5203.937702] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5203.941127] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5203.947308] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5204.451328] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5204.452424] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5204.460355] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5209.569607] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5209.574255] Lustre: Skipped 2 previous similar messages [ 5214.690335] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5214.694473] Lustre: Skipped 1 previous similar message [ 5215.200134] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5215.274253] Lustre: server umount lustre-OST0000 complete [ 5219.809933] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5219.812527] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5219.816858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5219.823610] Lustre: Skipped 1 previous similar message [ 5223.271756] Lustre: server umount lustre-MDT0000 complete [ 5224.929695] LustreError: 173558:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5224.943000] LustreError: 173558:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5225.292246] LustreError: 174956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772474723 with bad export cookie 9628781395752768566 [ 5225.299254] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5225.473573] Lustre: server umount lustre-MDT0001 complete [ 5230.570935] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 5231.864534] Key type lgssc unregistered [ 5232.009640] LNet: 176145:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5232.014170] LNetError: 176145:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5232.027383] LNet: Removed LNI 192.168.204.120@tcp [ 5232.422094] Key type .llcrypt unregistered [ 5232.423655] Key type ._llcrypt unregistered [ 5243.077403] Key type ._llcrypt registered [ 5243.078833] Key type .llcrypt registered [ 5243.121664] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 5243.513933] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5243.553550] alg: No test for adler32 (adler32-zlib) [ 5244.436180] Lustre: Lustre: Build Version: 2.17.50_194_gae5720b [ 5244.536842] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 5246.128388] Key type lgssc registered [ 5246.700416] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5250.145145] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5250.157774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5251.486935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5253.442156] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5257.704121] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5257.916384] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5259.768118] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5262.447148] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5264.195964] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5267.612465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5267.781995] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5270.697791] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5273.748590] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5277.045100] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 5282.771087] Lustre: Failing over lustre-MDT0000 [ 5282.926635] Lustre: server umount lustre-MDT0000 complete [ 5283.810979] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5283.817090] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5283.825035] LustreError: 177519:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5283.834317] LustreError: 177519:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5286.278024] LustreError: 179621:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5287.395866] LustreError: 177518:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5287.397121] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5291.380174] LustreError: 179621:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5291.394981] LustreError: 179621:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5296.496247] LustreError: 179675:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5296.510301] LustreError: 179675:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5297.685431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5297.777284] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5297.907175] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5297.939194] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5299.947903] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5301.615650] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5302.371736] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 5303.272372] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5303.288352] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 5303.318380] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 5308.097460] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 5313.508065] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5313.508427] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5313.514101] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5313.514887] LustreError: Skipped 1 previous similar message [ 5313.528620] Lustre: Skipped 2 previous similar messages [ 5318.627552] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5318.631912] Lustre: Skipped 1 previous similar message [ 5319.605548] Lustre: server umount lustre-OST0000 complete [ 5323.748182] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5323.756763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5323.766172] Lustre: Skipped 2 previous similar messages [ 5327.954158] Lustre: server umount lustre-MDT0000 complete [ 5328.865818] LustreError: 178626:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5328.875508] LustreError: 178626:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5330.308258] LustreError: 177498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772474828 with bad export cookie 6433791670185554265 [ 5330.311525] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5330.317203] LustreError: 177498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5330.537683] Lustre: server umount lustre-MDT0001 complete [ 5336.722352] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 5338.363826] Key type lgssc unregistered [ 5338.541520] LNet: 182035:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5338.545278] LNetError: 182035:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5338.559103] LNet: Removed LNI 192.168.204.120@tcp [ 5339.020686] Key type .llcrypt unregistered [ 5339.022942] Key type ._llcrypt unregistered [ 5352.168160] Key type ._llcrypt registered [ 5352.169153] Key type .llcrypt registered [ 5352.224397] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 5352.745110] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5352.810556] alg: No test for adler32 (adler32-zlib) [ 5353.719891] Lustre: Lustre: Build Version: 2.17.50_194_gae5720b [ 5353.867692] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 5355.480162] Key type lgssc registered [ 5356.106705] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5360.037837] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5360.049432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5361.316684] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5363.242910] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5367.889422] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5368.172907] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5370.123623] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5373.068556] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5375.094493] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5378.731076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5378.985194] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5381.929947] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5384.828070] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5386.161124] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 5392.826734] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 5395.494042] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 5403.269268] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5411.809652] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5411.815234] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5411.822784] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5413.859762] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5413.859817] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5413.867416] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5413.869649] Lustre: Skipped 1 previous similar message [ 5417.491317] Lustre: server umount lustre-OST0000 complete [ 5419.489384] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5419.494200] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5419.502649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5424.098655] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5424.103402] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5424.109325] Lustre: Skipped 1 previous similar message [ 5425.584888] Lustre: server umount lustre-MDT0000 complete [ 5427.405732] LustreError: 183389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772474925 with bad export cookie 2146142567370874375 [ 5427.413416] LustreError: 183389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5427.419161] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5427.584748] Lustre: server umount lustre-MDT0001 complete [ 5432.413864] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 5433.760647] Key type lgssc unregistered [ 5433.909430] LNet: 186753:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5433.912860] LNetError: 186753:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5433.924763] LNet: Removed LNI 192.168.204.120@tcp [ 5434.294818] Key type .llcrypt unregistered [ 5434.296137] Key type ._llcrypt unregistered [ 5445.833627] Key type ._llcrypt registered [ 5445.835064] Key type .llcrypt registered [ 5445.892531] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 5446.403310] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5446.457497] alg: No test for adler32 (adler32-zlib) [ 5447.343278] Lustre: Lustre: Build Version: 2.17.50_194_gae5720b [ 5447.454082] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 5449.064188] Key type lgssc registered [ 5449.606658] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5453.295411] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5453.307198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5454.527175] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5456.383552] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5460.162270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5460.331519] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5461.911194] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5464.475149] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5465.986283] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5469.101122] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5469.316178] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5472.166624] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5474.923710] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5479.604528] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 5482.155554] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 5490.658208] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5500.026957] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 5506.150319] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 5515.745141] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5515.748966] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5515.749750] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5515.760196] Lustre: Skipped 1 previous similar message [ 5519.247921] Lustre: server umount lustre-OST0000 complete [ 5521.890410] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5521.894883] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5521.900664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5521.902868] Lustre: Skipped 1 previous similar message [ 5525.985063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5525.986134] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5527.483534] Lustre: server umount lustre-MDT0000 complete [ 5529.478212] LustreError: 188106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772475027 with bad export cookie 14646625736156511581 [ 5529.481679] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5529.484942] LustreError: 188106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5529.634927] Lustre: server umount lustre-MDT0001 complete [ 5534.505798] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 5535.956682] Key type lgssc unregistered [ 5536.136507] LNet: 192980:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5536.141474] LNetError: 192980:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5536.159731] LNet: Removed LNI 192.168.204.120@tcp [ 5536.604859] Key type .llcrypt unregistered [ 5536.606487] Key type ._llcrypt unregistered [ 5548.176740] Key type ._llcrypt registered [ 5548.178109] Key type .llcrypt registered [ 5548.224185] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 5548.737151] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5548.745531] alg: No test for adler32 (adler32-zlib) [ 5549.640888] Lustre: Lustre: Build Version: 2.17.50_194_gae5720b [ 5549.768901] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 5551.393679] Key type lgssc registered [ 5552.066137] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5555.984431] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5555.993752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5557.229422] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5559.048074] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5562.748044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5562.921668] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5564.691778] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5567.412569] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5568.986583] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5572.008994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5572.068335] LustreError: 195946:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5572.074174] LustreError: 195946:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5572.173159] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5574.308574] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5576.731436] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5583.724781] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 5594.085793] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5594.087398] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5594.087515] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5594.095318] Lustre: Skipped 1 previous similar message [ 5594.108925] Lustre: Skipped 1 previous similar message [ 5598.756805] Lustre: server umount lustre-OST0000 complete [ 5603.811136] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5603.816535] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5603.825744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5606.642243] Lustre: server umount lustre-MDT0000 complete [ 5608.581064] LustreError: 195974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772475106 with bad export cookie 5378485084272256563 [ 5608.581659] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5608.585128] LustreError: 195974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5608.712696] Lustre: server umount lustre-MDT0001 complete [ 5614.286337] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 5615.817613] Key type lgssc unregistered [ 5615.976591] LNet: 197705:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5615.980976] LNetError: 197705:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5615.992074] LNet: Removed LNI 192.168.204.120@tcp [ 5616.404184] Key type .llcrypt unregistered [ 5616.407845] Key type ._llcrypt unregistered [ 5625.131252] Key type ._llcrypt registered [ 5625.132606] Key type .llcrypt registered [ 5625.180378] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 5625.573248] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5625.623292] alg: No test for adler32 (adler32-zlib) [ 5626.554565] Lustre: Lustre: Build Version: 2.17.50_194_gae5720b [ 5626.654241] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 5628.248193] Key type lgssc registered [ 5628.733834] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5632.618706] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5632.631376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5633.873480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5635.725476] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5640.284385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5640.496922] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5642.171489] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5644.554528] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5646.347297] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5649.445992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5649.612980] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5652.149890] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5654.650385] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5658.860659] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 5664.228412] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5664.229735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5664.233443] Lustre: Skipped 1 previous similar message [ 5666.077075] Lustre: server umount lustre-OST0000 complete [ 5669.348639] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5669.360286] Lustre: Skipped 1 previous similar message [ 5669.360479] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5669.368621] Lustre: Skipped 2 previous similar messages [ 5670.054229] Lustre: server umount lustre-MDT0000 complete [ 5671.688547] LustreError: 199649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772475169 with bad export cookie 17800604873249101292 [ 5671.691079] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5671.694944] LustreError: 199649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5671.827726] Lustre: server umount lustre-MDT0001 complete [ 5677.037247] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 5678.379541] Key type lgssc unregistered [ 5678.534658] LNet: 202255:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5678.539271] LNetError: 202255:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5678.551291] LNet: Removed LNI 192.168.204.120@tcp [ 5678.964222] Key type .llcrypt unregistered [ 5678.965628] Key type ._llcrypt unregistered [ 5687.732075] Key type ._llcrypt registered [ 5687.733329] Key type .llcrypt registered [ 5687.777469] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 5688.184431] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5688.230902] alg: No test for adler32 (adler32-zlib) [ 5689.121871] Lustre: Lustre: Build Version: 2.17.50_194_gae5720b [ 5689.234667] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 5690.840900] Key type lgssc registered [ 5691.305347] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5694.636424] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5694.647250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5695.926709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5697.532972] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5701.255111] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5701.459536] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5703.209822] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5705.642608] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5707.176279] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5709.894386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5710.095326] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5712.784504] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5715.301177] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5720.360744] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 5723.418884] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5723.451831] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5723.490127] LustreError: 205072:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5723.498596] LustreError: 205072:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5723.546916] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5725.774751] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5728.194030] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5729.923119] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 5740.406469] Lustre: Permanently deactivating lustre-OST0001 [ 5740.415445] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 5748.223533] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 5753.042508] Lustre: Permanently reactivating lustre-OST0001 [ 5753.049207] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 5753.052730] Lustre: Skipped 2 previous similar messages [ 5755.201694] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5755.207286] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 5755.210209] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5755.214678] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5755.235974] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5765.602256] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5765.603274] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5765.609405] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5765.613194] Lustre: Skipped 2 previous similar messages [ 5765.617846] Lustre: Skipped 1 previous similar message [ 5767.682466] Lustre: server umount lustre-OST0001 complete [ 5770.723838] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5770.724110] LustreError: 206586:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5770.725306] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5770.728346] Lustre: Skipped 1 previous similar message [ 5770.734783] LustreError: 206586:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5775.318197] Lustre: server umount lustre-OST0000 complete [ 5778.400987] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5778.403942] LustreError: Skipped 1 previous similar message [ 5778.405961] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5778.410792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5778.413433] Lustre: Skipped 1 previous similar message [ 5780.961223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5780.963498] Lustre: Skipped 1 previous similar message [ 5782.814661] Lustre: server umount lustre-MDT0000 complete [ 5784.237175] LustreError: 203462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772475282 with bad export cookie 16524349307508136579 [ 5784.240177] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5784.242181] LustreError: 203462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5784.356267] Lustre: server umount lustre-MDT0001 complete [ 5788.390986] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 5789.565418] Key type lgssc unregistered [ 5789.701429] LNet: 208055:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5789.704284] LNetError: 208055:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5789.715380] LNet: Removed LNI 192.168.204.120@tcp [ 5790.020376] Key type .llcrypt unregistered [ 5790.022341] Key type ._llcrypt unregistered [ 5805.396559] Key type ._llcrypt registered [ 5805.398052] Key type .llcrypt registered [ 5805.445307] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 5805.793077] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5805.810442] alg: No test for adler32 (adler32-zlib) [ 5806.683686] Lustre: Lustre: Build Version: 2.17.50_194_gae5720b [ 5806.775747] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 5808.360179] Key type lgssc registered [ 5808.788045] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5812.047643] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5812.059403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5813.203795] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5813.218227] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5813.303918] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5814.891144] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5818.324518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5818.386564] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5818.399221] Lustre: 210233:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5818.538370] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5820.064785] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5822.345687] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5823.769669] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5826.501350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5826.591665] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5826.690749] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5827.491982] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 5829.098140] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5831.305169] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5833.034558] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 5839.770731] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 5848.674349] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 5856.549517] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5869.831398] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5869.834612] Lustre: Skipped 1 previous similar message [ 5891.084055] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 5891.088063] Lustre: Skipped 2 previous similar messages [ 5926.960362] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5926.964762] Lustre: Skipped 4 previous similar messages [ 5994.518393] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 5994.521839] Lustre: Skipped 7 previous similar messages [ 6001.636554] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6001.637377] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6001.639576] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6001.647262] Lustre: Skipped 1 previous similar message [ 6002.750121] Lustre: server umount lustre-OST0000 complete [ 6006.754190] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6006.756047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6006.761809] Lustre: Skipped 1 previous similar message [ 6006.765852] Lustre: Skipped 2 previous similar messages [ 6010.467375] Lustre: server umount lustre-MDT0000 complete [ 6012.049820] LustreError: 212460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772475510 with bad export cookie 8712336831308008018 [ 6012.055363] LustreError: 212460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6012.056614] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6012.195333] Lustre: server umount lustre-MDT0001 complete [ 6016.870418] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 6018.448310] Key type lgssc unregistered [ 6018.620901] LNet: 214669:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6018.623225] LNetError: 214669:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6018.632345] LNet: Removed LNI 192.168.204.120@tcp [ 6019.004709] Key type .llcrypt unregistered [ 6019.006560] Key type ._llcrypt unregistered [ 6030.054640] Key type ._llcrypt registered [ 6030.055942] Key type .llcrypt registered [ 6030.105121] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 6030.719707] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6030.731390] alg: No test for adler32 (adler32-zlib) [ 6031.653751] Lustre: Lustre: Build Version: 2.17.50_194_gae5720b [ 6031.763303] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 6033.360276] Key type lgssc registered [ 6033.881279] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6037.698548] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6037.709315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6038.936892] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6040.831347] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6044.558228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6044.721775] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6046.424528] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6048.852148] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6050.381136] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6053.461944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6053.627075] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6056.358730] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6058.669671] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6063.849874] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 6079.465495] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6079.468133] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6079.471751] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6079.485841] Lustre: Skipped 1 previous similar message [ 6081.690077] Lustre: server umount lustre-OST0000 complete [ 6084.577665] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6084.581965] Lustre: Skipped 1 previous similar message [ 6084.582210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6084.590016] Lustre: Skipped 2 previous similar messages [ 6089.570268] Lustre: server umount lustre-MDT0000 complete [ 6089.696912] LustreError: 216045:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6089.703411] LustreError: 216045:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6091.057213] LustreError: 218285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772475589 with bad export cookie 5105516033668043060 [ 6091.062837] LustreError: 218285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6091.063240] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6091.189253] Lustre: server umount lustre-MDT0001 complete [ 6095.211345] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 6096.354269] Key type lgssc unregistered [ 6096.494397] LNet: 219376:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6096.498537] LNetError: 219376:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6096.507312] LNet: Removed LNI 192.168.204.120@tcp [ 6096.887577] Key type .llcrypt unregistered [ 6096.889087] Key type ._llcrypt unregistered [ 6107.020425] Key type ._llcrypt registered [ 6107.021575] Key type .llcrypt registered [ 6107.061120] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 6107.443145] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6107.518587] alg: No test for adler32 (adler32-zlib) [ 6108.381257] Lustre: Lustre: Build Version: 2.17.50_194_gae5720b [ 6108.484240] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 6110.072112] Key type lgssc registered [ 6110.500037] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6374.249850] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 6375.261362] Key type lgssc unregistered [ 6375.385496] LNet: 221342:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6375.390255] LNetError: 221342:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6375.403620] LNet: Removed LNI 192.168.204.120@tcp [ 6375.706264] Key type .llcrypt unregistered [ 6375.707809] Key type ._llcrypt unregistered [ 6384.335166] Key type ._llcrypt registered [ 6384.336751] Key type .llcrypt registered [ 6384.381088] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 6390.610357] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 6390.980889] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6391.015103] alg: No test for adler32 (adler32-zlib) [ 6391.871845] Lustre: Lustre: Build Version: 2.17.50_194_gae5720b [ 6391.955643] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 6393.536146] Key type lgssc registered [ 6393.873359] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6403.747454] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 6065 sec ========== 13:25:01 (1772475901) [ 6404.281939] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 13:25:01 (1772475901) === [ 6405.392679] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 13:25:02 (1772475902) === [ 6418.690664] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 6419.725257] Key type lgssc unregistered [ 6419.855621] LNet: 226494:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6419.858807] LNetError: 226494:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6419.870319] LNet: Removed LNI 192.168.204.120@tcp [ 6420.158541] Key type .llcrypt unregistered [ 6420.160352] Key type ._llcrypt unregistered