[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 510697823 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.003322] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.007742] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009010] pid_max: default: 32768 minimum: 301 [ 0.010157] LSM: Security Framework initializing [ 0.011053] Yama: becoming mindful. [ 0.012035] SELinux: Initializing. [ 0.013068] *** VALIDATE selinux *** [ 0.021783] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025829] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026159] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027105] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028115] *** VALIDATE tmpfs *** [ 0.030363] *** VALIDATE proc *** [ 0.031209] *** VALIDATE cgroup *** [ 0.032007] *** VALIDATE cgroup2 *** [ 0.033209] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034143] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035005] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036021] Spectre V2 : User space: Vulnerable [ 0.037006] Speculative Store Bypass: Vulnerable [ 0.039364] debug: unmapping init [mem 0xffffffffa5659000-0xffffffffa5660fff] [ 0.041911] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042534] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043019] ... version: 2 [ 0.043862] ... bit width: 48 [ 0.044011] ... generic registers: 4 [ 0.044953] ... value mask: 0000ffffffffffff [ 0.045011] ... max period: 00007fffffffffff [ 0.046010] ... fixed-purpose events: 3 [ 0.047009] ... event mask: 000000070000000f [ 0.048279] rcu: Hierarchical SRCU implementation. [ 0.050483] smp: Bringing up secondary CPUs ... [ 0.051624] x86: Booting SMP configuration: [ 0.052027] .... node #0, CPUs: #1 #2 #3 [ 0.055724] smp: Brought up 1 node, 4 CPUs [ 0.057016] smpboot: Max logical packages: 1 [ 0.058022] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.249032] node 0 deferred pages initialised in 188ms [ 0.252470] devtmpfs: initialized [ 0.253321] x86/mm: Memory block size: 128MB [ 0.257133] gcov: version magic: 0x41383552 [ 0.260383] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.261092] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.262270] pinctrl core: initialized pinctrl subsystem [ 0.263212] [ 0.263672] ************************************************************* [ 0.264022] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.265021] ** ** [ 0.266018] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.267020] ** ** [ 0.268014] ** This means that this kernel is built to expose internal ** [ 0.269016] ** IOMMU data structures, which may compromise security on ** [ 0.270027] ** your system. ** [ 0.271018] ** ** [ 0.272019] ** If you see this message and you are not debugging the ** [ 0.273014] ** kernel, report this immediately to your vendor! ** [ 0.274019] ** ** [ 0.275020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.276016] ************************************************************* [ 0.277983] NET: Registered protocol family 16 [ 0.279507] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.282071] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.284061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.286010] cpuidle: using governor menu [ 0.286820] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.289508] PCI: Using configuration type 1 for base access [ 0.290125] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.298101] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.299036] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.302081] cryptd: max_cpu_qlen set to 1000 [ 0.304219] ACPI: Added _OSI(Module Device) [ 0.305013] ACPI: Added _OSI(Processor Device) [ 0.306018] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.308022] ACPI: Added _OSI(Processor Aggregator Device) [ 0.312334] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.317311] ACPI: Interpreter enabled [ 0.319101] ACPI: PM: (supports S0 S3 S4 S5) [ 0.321037] ACPI: Using IOAPIC for interrupt routing [ 0.322237] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.327481] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.338014] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.340052] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.342028] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.346117] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.350492] acpiphp: Slot [2] registered [ 0.352210] acpiphp: Slot [5] registered [ 0.353112] acpiphp: Slot [6] registered [ 0.354207] acpiphp: Slot [7] registered [ 0.355091] acpiphp: Slot [8] registered [ 0.356031] acpiphp: Slot [9] registered [ 0.356975] acpiphp: Slot [10] registered [ 0.358618] acpiphp: Slot [3] registered [ 0.359075] acpiphp: Slot [4] registered [ 0.359828] acpiphp: Slot [11] registered [ 0.361062] acpiphp: Slot [12] registered [ 0.362080] acpiphp: Slot [13] registered [ 0.363113] acpiphp: Slot [14] registered [ 0.363884] acpiphp: Slot [15] registered [ 0.365079] acpiphp: Slot [16] registered [ 0.366010] acpiphp: Slot [17] registered [ 0.366969] acpiphp: Slot [18] registered [ 0.368083] acpiphp: Slot [19] registered [ 0.369156] acpiphp: Slot [20] registered [ 0.369996] acpiphp: Slot [21] registered [ 0.370065] acpiphp: Slot [22] registered [ 0.371094] acpiphp: Slot [23] registered [ 0.371958] acpiphp: Slot [24] registered [ 0.373060] acpiphp: Slot [25] registered [ 0.373860] acpiphp: Slot [26] registered [ 0.375101] acpiphp: Slot [27] registered [ 0.375957] acpiphp: Slot [28] registered [ 0.376059] acpiphp: Slot [29] registered [ 0.376919] acpiphp: Slot [30] registered [ 0.378058] acpiphp: Slot [31] registered [ 0.378971] PCI host bridge to bus 0000:00 [ 0.378978] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.381034] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.383026] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.385024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.387030] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.389027] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.390155] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.391865] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.394390] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.403019] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.407535] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.410038] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.413027] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.415025] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.418566] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.420724] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.422194] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.424710] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.429022] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.441016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.444019] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.448258] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.453020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.459022] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.480037] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.490718] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.501021] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.511031] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.532023] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.545184] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.555020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.562020] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.574017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.584096] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.596024] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.604022] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.618020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.628594] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.639028] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.645026] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.658032] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.672372] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.682018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.689019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.711021] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.724451] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.727367] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.730348] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.731284] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.733199] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.739151] iommu: Default domain type: Passthrough [ 0.741587] SCSI subsystem initialized [ 0.743148] ACPI: bus type USB registered [ 0.745122] usbcore: registered new interface driver usbfs [ 0.747134] usbcore: registered new interface driver hub [ 0.749128] usbcore: registered new device driver usb [ 0.751164] pps_core: LinuxPPS API ver. 1 registered [ 0.753014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.757093] PTP clock support registered [ 0.759061] EDAC MC: Ver: 3.0.0 [ 0.761198] PCI: Using ACPI for IRQ routing [ 0.762650] NetLabel: Initializing [ 0.764016] NetLabel: domain hash size = 128 [ 0.766013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.768096] NetLabel: unlabeled traffic allowed by default [ 0.770292] vgaarb: loaded [ 0.772007] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.773010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.782877] clocksource: Switched to clocksource kvm-clock [ 0.892295] VFS: Disk quotas dquot_6.6.0 [ 0.894243] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.896354] *** VALIDATE ramfs *** [ 0.897282] *** VALIDATE hugetlbfs *** [ 0.899023] pnp: PnP ACPI init [ 0.901613] pnp: PnP ACPI: found 6 devices [ 0.920537] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.923678] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.925855] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.927767] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.929831] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.932226] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.935133] NET: Registered protocol family 2 [ 0.937353] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.942437] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.947078] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.953436] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.957290] TCP: Hash tables configured (established 65536 bind 65536) [ 0.960860] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.964174] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.967507] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.970851] NET: Registered protocol family 1 [ 0.973560] RPC: Registered named UNIX socket transport module. [ 0.975912] RPC: Registered udp transport module. [ 0.977736] RPC: Registered tcp transport module. [ 0.979663] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.982547] NET: Registered protocol family 44 [ 0.984635] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.988371] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.990791] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.993344] PCI: CLS 0 bytes, default 64 [ 0.995170] Unpacking initramfs... [ 2.416334] debug: unmapping init [mem 0xffff9cd8fcc54000-0xffff9cd8fffbffff] [ 2.420280] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.422489] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.424861] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.928942] Initialise system trusted keyrings [ 2.930656] Key type blacklist registered [ 2.932966] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.943412] zbud: loaded [ 2.946244] *** VALIDATE nfs *** [ 2.947618] *** VALIDATE nfs4 *** [ 2.949329] pstore: using deflate compression [ 2.952797] Platform Keyring initialized [ 3.047823] NET: Registered protocol family 38 [ 3.050081] Key type asymmetric registered [ 3.051306] Asymmetric key parser 'x509' registered [ 3.052807] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.054950] io scheduler mq-deadline registered [ 3.056591] io scheduler kyber registered [ 3.058098] io scheduler bfq registered [ 3.060343] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.063369] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.066330] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.069290] ACPI: Power Button [PWRF] [ 3.073825] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.078648] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.089169] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.093950] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.104308] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.129907] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.156879] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.160520] Non-volatile memory driver v1.3 [ 3.161849] Linux agpgart interface v0.103 [ 3.199103] virtio_blk virtio1: [vda] 146536 512-byte logical blocks (75.0 MB/71.6 MiB) [ 3.202357] vda: detected capacity change from 0 to 75026432 [ 3.218663] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.221310] vdb: detected capacity change from 0 to 1073741824 [ 3.236664] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.239301] vdc: detected capacity change from 0 to 2621440000 [ 3.251442] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.253634] vdd: detected capacity change from 0 to 2621440000 [ 3.266046] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.269296] vde: detected capacity change from 0 to 4294967296 [ 3.287990] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.291147] vdf: detected capacity change from 0 to 4294967296 [ 3.299816] libphy: Fixed MDIO Bus: probed [ 3.309133] usbcore: registered new interface driver usbserial_generic [ 3.310951] usbserial: USB Serial support registered for generic [ 3.312773] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.316044] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.317285] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.319337] mousedev: PS/2 mouse device common for all mice [ 3.321481] rtc_cmos 00:05: RTC can wake from S4 [ 3.323580] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.324692] rtc_cmos 00:05: registered as rtc0 [ 3.327817] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.329612] intel_pstate: CPU model not supported [ 3.332424] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.334217] hid: raw HID events driver (C) Jiri Kosina [ 3.337390] usbcore: registered new interface driver usbhid [ 3.337463] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.338913] usbhid: USB HID core driver [ 3.342525] drop_monitor: Initializing network drop monitor service [ 3.344263] Initializing XFRM netlink socket [ 3.345602] NET: Registered protocol family 10 [ 3.347751] Segment Routing with IPv6 [ 3.348667] NET: Registered protocol family 17 [ 3.350346] mpls_gso: MPLS GSO support [ 3.354447] RAS: Correctable Errors collector initialized. [ 3.356164] AVX version of gcm_enc/dec engaged. [ 3.357254] AES CTR mode by8 optimization enabled [ 3.425060] sched_clock: Marking stable (3425009222, 0)->(4385738546, -960729324) [ 3.428576] registered taskstats version 1 [ 3.430802] Loading compiled-in X.509 certificates [ 3.432811] zswap: loaded using pool lzo/zbud [ 3.458868] Key type big_key registered [ 3.469450] Key type encrypted registered [ 3.470988] ima: No TPM chip found, activating TPM-bypass! [ 3.472524] ima: Allocated hash algorithm: sha1 [ 3.473701] ima: No architecture policies found [ 3.475060] evm: Initialising EVM extended attributes: [ 3.476316] evm: security.selinux [ 3.477378] evm: security.ima [ 3.478189] evm: security.capability [ 3.479091] evm: HMAC attrs: 0x1 [ 3.480822] rtc_cmos 00:05: setting system clock to 2026-08-22 02:46:14 UTC (1787366774) [ 3.485402] debug: unmapping init [mem 0xffffffffa6603000-0xffffffffa67fffff] [ 3.487524] debug: unmapping init [mem 0xffffffffa5382000-0xffffffffa5658fff] [ 3.493197] Write protecting the kernel read-only data: 28672k [ 3.495526] debug: unmapping init [mem 0xffffffffa3a03000-0xffffffffa3bfffff] [ 3.497525] debug: unmapping init [mem 0xffffffffa4314000-0xffffffffa43fffff] [ 3.524514] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.530185] systemd[1]: Detected virtualization kvm. [ 3.531359] systemd[1]: Detected architecture x86-64. [ 3.532497] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.558243] systemd[1]: No hostname configured. [ 3.559457] systemd[1]: Set hostname to . [ 3.560852] random: systemd: uninitialized urandom read (16 bytes read) [ 3.562546] systemd[1]: Initializing machine ID from random generator. [ 3.618951] random: ln: uninitialized urandom read (6 bytes read) [ 3.711546] random: systemd: uninitialized urandom read (16 bytes read) [ 3.713395] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.717233] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.720955] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Timers. [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Setup Virtual Console... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.368951] device-mapper: uevent: version 1.0.3 [ 4.371608] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.140322] virtio_net virtio0 ens2: renamed from eth0 [ 5.153930] random: fast init done [ 5.202069] scsi host0: ata_piix [ 5.222756] scsi host1: ata_piix [ 5.237516] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.240129] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.965129] dracut-initqueue[581]: RTNETLINK answers: File exists [ 9.985997] random: crng init done [ 9.986902] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.525377] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.774896] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.022812] SELinux: Disabled at runtime. [ 12.084190] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.089702] systemd[1]: Detected virtualization kvm. [ 12.090902] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.569517] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.573418] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.578857] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.584385] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.587533] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.595061] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.601030] systemd[1]: Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting POSIX Message Queue File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Create list of required st…ce nodes for the current kernel... Activating swap /dev/disk/by-label/SWAP... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice User and Session Slice. Starting Apply Kernel Variables... [ OK ] Created slice system-serial\x2dg[ 12.676727] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS etty.slice. [ OK ] Reached target RPC Port Mapper. Mounting Huge Pages File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target Paths. [ OK ] Stopped target Initrd File Systems. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-getty.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target Local Encrypted Volumes. Mounting Kernel Debug File System... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 13.075460] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.401248] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.454684] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.560122] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.585556] EDAC sbridge: Ver: 1.1.2 [ 15.254700] Key type dns_resolver registered [ 15.568106] NFS: Registering the id_resolver key type [ 15.570234] Key type id_resolver registered [ 15.571976] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg151-server login: [ 41.746413] libcfs: loading out-of-tree module taints kernel. [ 41.773110] Key type ._llcrypt registered [ 41.774297] Key type .llcrypt registered [ 41.832301] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_hostid [ 50.742455] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 51.341122] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 51.349472] alg: No test for adler32 (adler32-zlib) [ 52.368330] Lustre: Lustre: Build Version: 2.17.57_45_gb800aec [ 52.720044] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 54.351196] Key type lgssc registered [ 56.599801] Lustre: Echo OBD driver; http://www.lustre.org/ [ 72.026388] hrtimer: interrupt took 5497245 ns [ 80.603570] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 133.770330] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 149.163606] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 149.202775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 150.589358] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 150.616975] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 150.704640] Lustre: lustre-MDT0000: new disk, initializing [ 150.900468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 150.940186] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 156.917836] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 173.747523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 173.931501] Lustre: 6501:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 173.961073] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 173.964636] Lustre: Skipped 1 previous similar message [ 174.069726] Lustre: lustre-MDT0001: new disk, initializing [ 174.179235] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 174.205138] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 174.214961] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 178.946605] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 183.926517] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 193.823316] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 194.161990] Lustre: lustre-OST0000: new disk, initializing [ 194.166901] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 194.180791] Lustre: 8440:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 194.262799] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 201.133804] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 201.233428] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 201.239776] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 201.341906] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 216.215602] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 216.349163] Lustre: lustre-OST0001: new disk, initializing [ 216.355556] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 216.360378] Lustre: 9512:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 216.415571] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 223.029115] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 225.301694] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 225.309183] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 225.391903] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 236.145807] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 242.764406] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 249.244432] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing check_logdir /tmp/testlogs/ [ 254.939455] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing yml_node [ 260.265963] Lustre: DEBUG MARKER: Client: 2.17.57.45 [ 262.808768] Lustre: DEBUG MARKER: MDS: 2.17.57.45 [ 265.370601] Lustre: DEBUG MARKER: OSS: 2.17.57.45 [ 267.457604] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Aug 21 22:50:37 EDT 2026 [ 287.030160] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 288.543149] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 307.173195] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 307.175766] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 307.189213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 307.197965] Lustre: Skipped 2 previous similar messages [ 310.778870] Lustre: server umount lustre-MDT0000 complete [ 314.515385] LustreError: 8450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787367085 with bad export cookie 9063936651997975121 [ 314.530828] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 314.539676] LustreError: 8450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 317.409731] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 317.410563] LustreError: 6508:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 317.423206] Lustre: Skipped 1 previous similar message [ 317.447092] LustreError: 6508:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 317.457334] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 317.461124] Lustre: Skipped 4 previous similar messages [ 319.459075] LustreError: 6508:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 319.459214] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 319.486637] LustreError: 6508:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 319.503961] Lustre: Skipped 1 previous similar message [ 321.035162] Lustre: server umount lustre-MDT0001 complete [ 325.257102] Lustre: server umount lustre-OST0000 complete [ 329.000766] Lustre: server umount lustre-OST0001 complete [ 334.870545] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_hostid [ 344.322631] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 383.710177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 384.153590] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 384.206799] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 384.311332] Lustre: lustre-MDT0000: new disk, initializing [ 384.390113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 384.413563] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 389.314174] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 399.692731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 399.802671] Lustre: 15067:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 399.862855] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 399.866164] Lustre: Skipped 1 previous similar message [ 400.009381] Lustre: lustre-MDT0001: new disk, initializing [ 400.164319] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 400.234056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 400.252262] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 406.067671] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 417.293698] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 423.953826] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 432.164884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 432.632769] Lustre: lustre-OST0000: new disk, initializing [ 432.635805] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 432.641534] Lustre: 17038:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 432.718960] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 434.669141] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 434.682784] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 434.783594] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 439.031474] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 449.688567] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 455.509590] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 455.763590] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 461.529632] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 461.806628] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 465.383101] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 465.383625] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 465.399748] LustreError: Skipped 1 previous similar message [ 465.415049] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 465.429050] Lustre: Skipped 2 previous similar messages [ 465.450442] Lustre: Skipped 1 previous similar message [ 470.504812] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 470.512084] Lustre: Skipped 1 previous similar message [ 478.175234] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 478.348285] Lustre: server umount lustre-OST0000 complete [ 482.272865] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 482.279063] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 482.289751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 482.303397] Lustre: Skipped 2 previous similar messages [ 487.904634] Lustre: server umount lustre-MDT0000 complete [ 490.977143] LustreError: 15076:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 491.008645] LustreError: 15076:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 491.131939] LustreError: 15061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787367262 with bad export cookie 9063936651997976654 [ 491.141194] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 491.144663] LustreError: 15061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 491.464748] Lustre: server umount lustre-MDT0001 complete [ 494.791703] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 22:54:24 (1787367264) [ 500.927586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 501.296784] LustreError: 19305:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 501.377954] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 505.554242] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 506.850206] LustreError: 19306:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 514.658992] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 519.662442] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 527.465913] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 534.896053] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 542.113600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 550.291834] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 559.492446] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 570.344151] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 570.356607] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 570.385748] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 570.405655] Lustre: Skipped 2 previous similar messages [ 570.421407] Lustre: Skipped 3 previous similar messages [ 573.832789] Lustre: server umount lustre-OST0000 complete [ 580.591586] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 580.611429] Lustre: Skipped 1 previous similar message [ 584.008341] Lustre: server umount lustre-MDT0000 complete [ 585.697374] LustreError: 19302:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 585.718402] LustreError: 19302:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 588.129619] LustreError: 19286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787367359 with bad export cookie 9063936651997977529 [ 588.137436] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 588.141099] LustreError: 19286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 588.452981] Lustre: server umount lustre-MDT0001 complete [ 600.287596] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 603.161747] Key type lgssc unregistered [ 603.518948] LNet: 22840:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 603.542325] LNetError: 22840:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 603.593536] LNet: Removed LNI 192.168.201.151@tcp [ 604.563172] Key type .llcrypt unregistered [ 604.568074] Key type ._llcrypt unregistered [ 629.089864] Key type ._llcrypt registered [ 629.093257] Key type .llcrypt registered [ 629.219767] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 630.474267] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 630.525154] alg: No test for adler32 (adler32-zlib) [ 631.597960] Lustre: Lustre: Build Version: 2.17.57_45_gb800aec [ 631.908499] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 633.599179] Key type lgssc registered [ 634.556199] Lustre: Echo OBD driver; http://www.lustre.org/ [ 641.722364] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 641.746078] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 643.317817] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 647.337391] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 653.804448] LustreError: 24255:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 653.837484] LustreError: 24255:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 656.732796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 657.191837] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 661.191477] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 668.709961] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 674.760847] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 681.726847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 682.088692] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 688.008557] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 688.258276] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 696.036403] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 712.075507] Lustre: server umount lustre-OST0000 complete [ 718.816323] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 718.823853] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 718.839691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 719.331892] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 724.450973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 724.460151] Lustre: Skipped 3 previous similar messages [ 726.724365] Lustre: server umount lustre-MDT0000 complete [ 729.572317] LustreError: 24254:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 729.587940] LustreError: 24254:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 730.440250] LustreError: 24234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787367501 with bad export cookie 2894783037678192354 [ 730.441464] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 730.449435] LustreError: 24234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 740.976723] Lustre: server umount lustre-MDT0001 complete [ 751.227995] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 753.606167] Key type lgssc unregistered [ 753.871559] LNet: 28240:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 753.875423] LNetError: 28240:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 753.902646] LNet: Removed LNI 192.168.201.151@tcp [ 754.710161] Key type .llcrypt unregistered [ 754.712270] Key type ._llcrypt unregistered [ 776.919871] Key type ._llcrypt registered [ 776.922764] Key type .llcrypt registered [ 777.016245] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 777.854233] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 777.947891] alg: No test for adler32 (adler32-zlib) [ 779.022210] Lustre: Lustre: Build Version: 2.17.57_45_gb800aec [ 779.205953] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 780.847167] Key type lgssc registered [ 781.852716] Lustre: Echo OBD driver; http://www.lustre.org/ [ 788.896939] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 788.915140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 790.260068] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 794.636855] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 800.743285] LustreError: 29655:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 800.763127] LustreError: 29655:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 803.274530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 803.641491] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 808.293895] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 816.406342] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 825.021927] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 839.899578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 840.298488] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 846.455578] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 854.775870] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 860.789547] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 871.406593] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 871.407875] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 871.412884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 871.431200] Lustre: Skipped 1 previous similar message [ 876.523089] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 876.532952] Lustre: Skipped 2 previous similar messages [ 876.984042] Lustre: server umount lustre-OST0000 complete [ 881.636232] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 881.640898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 881.660061] Lustre: Skipped 1 previous similar message [ 881.677029] Lustre: Skipped 1 previous similar message [ 886.752962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 886.763685] Lustre: Skipped 1 previous similar message [ 887.562351] Lustre: server umount lustre-MDT0000 complete [ 891.635041] LustreError: 29634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787367662 with bad export cookie 6061866440875546146 [ 891.645349] LustreError: 29634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 891.645614] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 892.086798] Lustre: server umount lustre-MDT0001 complete [ 903.647729] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 906.287473] Key type lgssc unregistered [ 906.583021] LNet: 33573:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 906.597049] LNetError: 33573:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 906.626665] LNet: Removed LNI 192.168.201.151@tcp [ 907.461650] Key type .llcrypt unregistered [ 907.464099] Key type ._llcrypt unregistered [ 932.343690] Key type ._llcrypt registered [ 932.349466] Key type .llcrypt registered [ 932.459406] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 933.590600] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 933.690531] alg: No test for adler32 (adler32-zlib) [ 934.835848] Lustre: Lustre: Build Version: 2.17.57_45_gb800aec [ 935.116830] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 936.855232] Key type lgssc registered [ 937.977909] Lustre: Echo OBD driver; http://www.lustre.org/ [ 945.751320] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 945.767034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 947.254801] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 952.239492] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 957.411522] LustreError: 34986:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 957.435865] LustreError: 34986:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 961.606323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 962.028794] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 967.040294] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 974.080517] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 980.412826] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 986.251887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 986.551381] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 992.615374] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 992.756794] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1000.038623] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1012.192859] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1012.206873] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1012.223744] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1013.218473] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1013.220221] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1015.777532] Lustre: server umount lustre-OST0000 complete [ 1023.467882] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1023.475841] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1023.479746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1023.502803] Lustre: Skipped 1 previous similar message [ 1023.514069] Lustre: Skipped 2 previous similar messages [ 1025.831775] Lustre: server umount lustre-MDT0000 complete [ 1028.580577] LustreError: 35714:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1028.612436] LustreError: 35714:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1030.362964] LustreError: 35721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787367801 with bad export cookie 5586573942034291566 [ 1030.377903] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1030.388621] LustreError: 35721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1030.963818] Lustre: server umount lustre-MDT0001 complete [ 1043.663839] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 1047.513477] Key type lgssc unregistered [ 1048.083336] LNet: 38526:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1048.111924] LNetError: 38526:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1048.138728] LNet: Removed LNI 192.168.201.151@tcp [ 1049.014104] Key type .llcrypt unregistered [ 1049.018506] Key type ._llcrypt unregistered [ 1075.209293] Key type ._llcrypt registered [ 1075.212656] Key type .llcrypt registered [ 1075.374667] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 1076.818708] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1076.873411] alg: No test for adler32 (adler32-zlib) [ 1078.029818] Lustre: Lustre: Build Version: 2.17.57_45_gb800aec [ 1078.295875] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 1079.967160] Key type lgssc registered [ 1081.139856] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1088.334843] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1088.356766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1089.856387] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1094.065846] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 1099.236107] LustreError: 39948:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1099.257784] LustreError: 39948:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1102.335772] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1102.653328] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1107.887025] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 1116.273571] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1122.576212] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1130.923748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1131.175824] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1134.259673] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1137.704450] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 1145.373817] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1153.011161] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1153.017908] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1153.055507] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1158.119907] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1158.124551] Lustre: Skipped 2 previous similar messages [ 1163.235680] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1163.242210] Lustre: Skipped 2 previous similar messages [ 1166.303119] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1166.439643] Lustre: server umount lustre-OST0000 complete [ 1188.833057] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1188.834461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1188.847855] Lustre: Skipped 2 previous similar messages [ 1188.851973] Lustre: Skipped 1 previous similar message [ 1191.170879] Lustre: server umount lustre-MDT0000 complete [ 1192.994269] LustreError: 39949:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1193.017190] LustreError: 39949:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1195.134423] LustreError: 39929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787367966 with bad export cookie 13403464502278166343 [ 1195.141978] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1195.142661] LustreError: 39929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1201.542896] Lustre: server umount lustre-MDT0001 complete [ 1214.544834] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 1217.145463] Key type lgssc unregistered [ 1217.442615] LNet: 43660:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1217.452152] LNetError: 43660:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1217.470910] LNet: Removed LNI 192.168.201.151@tcp [ 1218.206949] Key type .llcrypt unregistered [ 1218.213223] Key type ._llcrypt unregistered [ 1242.334489] Key type ._llcrypt registered [ 1242.337166] Key type .llcrypt registered [ 1242.527475] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 1243.775174] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1243.977753] alg: No test for adler32 (adler32-zlib) [ 1245.029746] Lustre: Lustre: Build Version: 2.17.57_45_gb800aec [ 1245.292567] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 1247.007495] Key type lgssc registered [ 1248.207221] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1255.591227] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1255.618965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1257.275868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1261.736435] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 1267.682051] LustreError: 45083:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1267.704362] LustreError: 45083:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1270.289544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1270.708751] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1274.912240] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 1281.578770] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1286.921441] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1293.217714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1293.519023] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1297.928747] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 1302.772071] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1303.976429] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1310.799849] Lustre: server umount lustre-MDT0000 complete [ 1311.712931] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1311.719244] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1311.733896] LustreError: 47122:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1311.759301] LustreError: 47122:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1313.250838] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1313.251796] LustreError: 45082:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1313.298902] LustreError: 45082:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1314.437587] LustreError: 45064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787368085 with bad export cookie 3989269338973979914 [ 1314.451600] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1316.838860] LustreError: 47148:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1316.869933] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.51@tcp (stopping) [ 1318.384468] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1318.404572] Lustre: Skipped 1 previous similar message [ 1318.414512] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1320.729340] Lustre: server umount lustre-MDT0001 complete [ 1341.404336] Lustre: server umount lustre-OST0000 complete [ 1358.889621] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 1361.339659] Key type lgssc unregistered [ 1361.557704] LNet: 48953:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1361.582682] LNetError: 48953:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1362.662769] LNet: Removed LNI 192.168.201.151@tcp [ 1363.526820] Key type .llcrypt unregistered [ 1363.535464] Key type ._llcrypt unregistered [ 1387.995648] Key type ._llcrypt registered [ 1387.997922] Key type .llcrypt registered [ 1388.070163] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 1389.256189] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1389.294195] alg: No test for adler32 (adler32-zlib) [ 1390.242701] Lustre: Lustre: Build Version: 2.17.57_45_gb800aec [ 1390.431281] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 1392.119128] Key type lgssc registered [ 1393.242881] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1400.298323] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1400.318202] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1415.903663] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1421.024067] LustreError: 50340:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.151@tcp: failed processing log, type 4: rc = -110 [ 1447.775961] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1453.510513] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 1460.909185] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1627.228301] Lustre: server umount lustre-OST0000 complete [ 1642.686863] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 1644.772825] Key type lgssc unregistered [ 1645.014705] LNet: 52017:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1645.019818] LNetError: 52017:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1645.033178] LNet: Removed LNI 192.168.201.151@tcp [ 1645.651987] Key type .llcrypt unregistered [ 1645.653793] Key type ._llcrypt unregistered [ 1669.786695] Key type ._llcrypt registered [ 1669.790530] Key type .llcrypt registered [ 1669.897826] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 1671.160391] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1671.228694] alg: No test for adler32 (adler32-zlib) [ 1672.409592] Lustre: Lustre: Build Version: 2.17.57_45_gb800aec [ 1672.782466] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 1674.455309] Key type lgssc registered [ 1675.568812] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1682.440200] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1682.453543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1683.829680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1688.365572] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 1693.154823] LustreError: 53434:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1693.166039] LustreError: 53434:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1696.772890] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1697.094409] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1701.593953] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 1708.940634] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1714.715351] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1720.711475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1721.047547] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1726.707696] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 1728.255226] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1733.869838] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1737.701177] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1737.712505] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1737.729622] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1738.722534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1738.733366] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1738.741132] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1742.375606] Lustre: server umount lustre-OST0000 complete [ 1748.456274] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1748.464942] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1748.478934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1748.486635] Lustre: Skipped 1 previous similar message [ 1752.755257] Lustre: server umount lustre-MDT0000 complete [ 1754.081586] LustreError: 53434:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1754.094377] LustreError: 53434:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1756.415972] LustreError: 53416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787368527 with bad export cookie 1518146117389519598 [ 1756.417646] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1756.426338] LustreError: 53416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1756.782680] Lustre: server umount lustre-MDT0001 complete [ 1768.662586] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 1771.423692] Key type lgssc unregistered [ 1771.709587] LNet: 56969:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1771.720070] LNetError: 56969:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1771.742026] LNet: Removed LNI 192.168.201.151@tcp [ 1772.875436] Key type .llcrypt unregistered [ 1772.878200] Key type ._llcrypt unregistered [ 1796.650011] Key type ._llcrypt registered [ 1796.652139] Key type .llcrypt registered [ 1796.777912] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 1797.573580] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1797.700739] alg: No test for adler32 (adler32-zlib) [ 1798.933195] Lustre: Lustre: Build Version: 2.17.57_45_gb800aec [ 1799.163650] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 1800.818787] Key type lgssc registered [ 1801.965208] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1810.122995] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1810.158343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1825.760100] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1830.879285] LustreError: 58354:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.151@tcp: failed processing log, type 4: rc = -110 [ 1857.504667] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1864.841493] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 1872.864656] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1879.702028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1880.305138] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1880.429298] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1884.570895] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 1885.690034] LustreError: 58380:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1885.714420] LustreError: 58380:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1890.791754] LustreError: 58379:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1892.852039] LustreError: 58380:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1892.878932] LustreError: 58380:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1892.891371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1893.231247] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1897.587493] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 1905.287136] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1912.171448] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1916.386794] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1916.407085] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1916.430875] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1918.950894] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1918.966533] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1918.974082] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1920.701975] Lustre: server umount lustre-OST0000 complete [ 1944.545153] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1944.555710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1944.565479] Lustre: Skipped 1 previous similar message [ 1944.569813] Lustre: Skipped 1 previous similar message [ 1944.577921] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1948.616617] Lustre: server umount lustre-MDT0000 complete [ 1949.674228] LustreError: 60433:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1949.707151] LustreError: 60433:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1952.520726] LustreError: 58360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787368723 with bad export cookie 14756930598183999795 [ 1952.526426] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1952.530592] LustreError: 58360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1959.155616] Lustre: server umount lustre-MDT0001 complete [ 1973.232961] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 1977.539932] Key type lgssc unregistered [ 1977.834665] LNet: 62100:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1977.843975] LNetError: 62100:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1977.885819] LNet: Removed LNI 192.168.201.151@tcp [ 1979.196165] Key type .llcrypt unregistered [ 1979.198514] Key type ._llcrypt unregistered [ 2003.950260] Key type ._llcrypt registered [ 2003.952168] Key type .llcrypt registered [ 2004.063511] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 2005.274147] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2005.337070] alg: No test for adler32 (adler32-zlib) [ 2006.344605] Lustre: Lustre: Build Version: 2.17.57_45_gb800aec [ 2006.562096] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 2008.207922] Key type lgssc registered [ 2009.799846] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2018.688850] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2018.737633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2020.342290] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2025.490426] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 2030.564531] LustreError: 63516:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2030.590674] LustreError: 63516:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2035.468514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2035.683182] LustreError: 63517:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2035.961301] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2040.303246] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 2049.455054] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2055.454939] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2062.522881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2062.943649] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2065.011713] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2070.536895] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 2079.187911] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2100.364220] Lustre: server umount lustre-OST0000 complete [ 2106.344156] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2106.345911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2106.364030] Lustre: Skipped 1 previous similar message [ 2106.369206] Lustre: Skipped 1 previous similar message [ 2111.457146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2111.472577] Lustre: Skipped 1 previous similar message [ 2113.617421] Lustre: server umount lustre-MDT0000 complete [ 2116.585487] LustreError: 63517:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2116.626074] LustreError: 63517:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2117.349138] LustreError: 63497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787368888 with bad export cookie 12732458851393932561 [ 2117.364973] LustreError: 63497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2117.365628] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2128.086604] Lustre: server umount lustre-MDT0001 complete [ 2143.970399] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 2146.825274] Key type lgssc unregistered [ 2147.143939] LNet: 67078:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2147.159303] LNetError: 67078:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2147.197854] LNet: Removed LNI 192.168.201.151@tcp [ 2148.106833] Key type .llcrypt unregistered [ 2148.110706] Key type ._llcrypt unregistered [ 2176.043751] Key type ._llcrypt registered [ 2176.046018] Key type .llcrypt registered [ 2176.173366] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 2177.494777] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2177.519942] alg: No test for adler32 (adler32-zlib) [ 2178.594911] Lustre: Lustre: Build Version: 2.17.57_45_gb800aec [ 2178.837845] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 2180.479258] Key type lgssc registered [ 2181.397313] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2190.683410] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 23:22:40 (1787368960) [ 2197.760585] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2197.779450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2199.213894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2203.617447] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 2209.784274] LustreError: 68785:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2209.801285] LustreError: 68785:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2213.282171] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2213.931057] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2218.538986] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 2227.194749] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2233.192433] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2240.616849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2241.096761] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2244.153148] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2247.887891] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 2255.876000] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2265.561235] Lustre: Failing over lustre-MDT0000 [ 2265.988771] Lustre: server umount lustre-MDT0000 complete [ 2268.173629] LustreError: 68781:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2268.652642] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2268.684281] Lustre: Skipped 1 previous similar message [ 2273.255654] LustreError: 68780:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2273.276296] LustreError: 68780:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2275.959107] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2276.086839] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2276.248856] Lustre: *** cfs_fail_loc=135, val=0*** [ 2276.363600] LustreError: 71872:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2276.372252] LustreError: 71872:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.151@tcp: cfg command failed: rc = -2 [ 2276.394506] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2276.394506] [ 2276.402510] LustreError: MGC192.168.201.151@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2276.413268] LustreError: 71860:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2276.423262] LustreError: 71860:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2276.439944] LustreError: 71860:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2276.449453] LustreError: 71860:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2276.491696] Lustre: server umount lustre-MDT0000 complete [ 2276.494373] LustreError: 71860:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2278.381445] LustreError: 68780:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2278.396105] LustreError: 68780:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2283.211618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2283.327345] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2283.341684] LustreError: 68776:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2283.351197] LustreError: 72217:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGC192.168.201.151@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (00000000ef95a50f) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 2283.380763] LustreError: 70289:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2283.403766] LustreError: 70289:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2283.511299] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.51@tcp (not set up) [ 2283.723929] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2283.758192] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2287.600866] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 2288.624649] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2289.162875] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2289.206308] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2289.276165] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2299.368826] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2299.391392] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2299.421665] Lustre: Skipped 1 previous similar message [ 2299.438860] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2302.153952] Lustre: server umount lustre-OST0000 complete [ 2309.601939] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2309.605607] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2309.609174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2309.609182] Lustre: Skipped 1 previous similar message [ 2309.613447] Lustre: Skipped 2 previous similar messages [ 2312.092962] Lustre: server umount lustre-MDT0000 complete [ 2314.720891] LustreError: 70289:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2314.745438] LustreError: 70289:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2315.743494] LustreError: 68765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787369086 with bad export cookie 1471115969254158373 [ 2315.753237] LustreError: 68765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2315.759689] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2316.485936] Lustre: server umount lustre-MDT0001 complete [ 2327.732645] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 2330.722799] Key type lgssc unregistered [ 2331.092796] LNet: 73659:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2331.098985] LNetError: 73659:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2331.111842] LNet: Removed LNI 192.168.201.151@tcp [ 2331.800941] Key type .llcrypt unregistered [ 2331.803264] Key type ._llcrypt unregistered [ 2355.160208] Key type ._llcrypt registered [ 2355.162864] Key type .llcrypt registered [ 2355.253753] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 2356.171542] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2356.281953] alg: No test for adler32 (adler32-zlib) [ 2357.382435] Lustre: Lustre: Build Version: 2.17.57_45_gb800aec [ 2357.713041] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 2359.439748] Key type lgssc registered [ 2361.091906] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2369.780106] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2369.799046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2371.306974] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2376.254633] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 2381.798784] LustreError: 75070:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2381.818762] LustreError: 75070:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2386.031537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2386.342971] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2390.811180] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 2399.376162] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2405.610209] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2412.691910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2413.147225] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2416.254108] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2420.884449] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 2429.160490] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2439.192591] Lustre: Failing over lustre-MDT0000 [ 2439.433529] Lustre: server umount lustre-MDT0000 complete [ 2442.261124] LustreError: 75066:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2442.728218] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2442.740723] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2443.242888] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2443.268045] LustreError: 76287:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2443.299925] LustreError: 76287:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2447.336380] LustreError: 76287:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2449.353532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2449.561269] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2449.744437] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2449.905296] LustreError: 78156:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2449.913809] LustreError: 78156:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.151@tcp: cfg command failed: rc = -9 [ 2449.930787] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2449.930787] [ 2449.946496] LustreError: MGC192.168.201.151@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2449.955493] LustreError: 78145:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2449.966121] LustreError: 78145:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 2449.980674] LustreError: 78145:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2449.987327] LustreError: 78145:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2450.035478] Lustre: server umount lustre-MDT0000 complete [ 2450.040154] LustreError: 78145:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2452.452890] LustreError: 75067:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2452.481534] LustreError: 75067:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2457.315067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2457.401481] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2457.406990] LustreError: 75062:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2457.572930] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.51@tcp (not set up) [ 2457.759402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2457.821519] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2462.698296] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2462.976538] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 2463.233766] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2463.265515] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2463.338506] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2473.462091] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2473.474675] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2473.475308] Lustre: Skipped 2 previous similar messages [ 2473.491054] Lustre: Skipped 1 previous similar message [ 2477.776822] Lustre: server umount lustre-OST0000 complete [ 2483.683721] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2483.685606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2483.688714] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2483.698188] Lustre: Skipped 1 previous similar message [ 2483.721058] Lustre: Skipped 1 previous similar message [ 2488.803163] LustreError: 75067:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2488.818054] LustreError: 75067:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2490.118850] Lustre: server umount lustre-MDT0000 complete [ 2494.532322] LustreError: 75052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787369265 with bad export cookie 13285467827937435530 [ 2494.533922] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2494.547353] LustreError: 75052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2495.182698] Lustre: server umount lustre-MDT0001 complete [ 2506.655044] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 2509.966483] Key type lgssc unregistered [ 2510.329254] LNet: 79944:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2510.339109] LNetError: 79944:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2510.363884] LNet: Removed LNI 192.168.201.151@tcp [ 2511.292806] Key type .llcrypt unregistered [ 2511.294723] Key type ._llcrypt unregistered [ 2526.033319] Key type ._llcrypt registered [ 2526.034986] Key type .llcrypt registered [ 2526.223337] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 2526.662727] Key type .llcrypt unregistered [ 2526.665555] Key type ._llcrypt unregistered [ 2555.828873] Key type ._llcrypt registered [ 2555.832171] Key type .llcrypt registered [ 2555.945734] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 2557.080527] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2557.249862] alg: No test for adler32 (adler32-zlib) [ 2558.371720] Lustre: Lustre: Build Version: 2.17.57_45_gb800aec [ 2558.595972] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 2560.343186] Key type lgssc registered [ 2561.779041] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2570.719798] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2570.741315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2572.209783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2577.066358] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 2582.514831] LustreError: 81885:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2582.533028] LustreError: 81885:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2585.277327] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2585.601831] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2590.754116] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 2599.729294] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2607.169628] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2615.403331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2615.873888] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2617.930538] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2624.478312] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 2635.262682] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2648.545750] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2648.559101] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2648.580149] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2650.593884] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2650.597802] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2650.599038] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2650.668190] Lustre: Skipped 1 previous similar message [ 2654.192978] Lustre: server umount lustre-OST0000 complete [ 2660.834788] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2660.841813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2660.851047] Lustre: Skipped 1 previous similar message [ 2660.881199] Lustre: Skipped 1 previous similar message [ 2665.671817] Lustre: server umount lustre-MDT0000 complete [ 2665.957600] LustreError: 81885:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2670.053769] LustreError: 81885:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2670.077116] LustreError: 81885:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2670.310644] LustreError: 81865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787369441 with bad export cookie 220167880678023731 [ 2670.316194] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2670.604705] Lustre: server umount lustre-MDT0001 complete [ 2685.296860] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 2688.654382] Key type lgssc unregistered [ 2688.929815] LNet: 85439:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2688.934843] LNetError: 85439:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2688.952513] LNet: Removed LNI 192.168.201.151@tcp [ 2689.821079] Key type .llcrypt unregistered [ 2689.824667] Key type ._llcrypt unregistered [ 2717.735931] Key type ._llcrypt registered [ 2717.743290] Key type .llcrypt registered [ 2717.815236] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 2718.845156] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2718.873087] alg: No test for adler32 (adler32-zlib) [ 2720.105764] Lustre: Lustre: Build Version: 2.17.57_45_gb800aec [ 2720.419243] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 2722.079790] Key type lgssc registered [ 2723.586471] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2731.752406] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2731.763184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2733.120196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2737.591737] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 2743.268175] LustreError: 86850:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2743.287575] LustreError: 86850:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2747.030593] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2747.550279] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2752.044561] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 2760.481593] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2767.244410] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2776.325777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2776.764477] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2778.831655] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2785.035670] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 2793.084979] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2802.658889] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2802.660079] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2802.676713] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2802.684438] Lustre: Skipped 1 previous similar message [ 2807.503662] Lustre: server umount lustre-OST0000 complete [ 2812.899532] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2812.907254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2812.926661] Lustre: Skipped 1 previous similar message [ 2812.935894] Lustre: Skipped 2 previous similar messages [ 2817.896188] Lustre: server umount lustre-MDT0000 complete [ 2818.018843] LustreError: 86846:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2818.045801] LustreError: 86846:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2822.122060] LustreError: 86851:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2822.165929] LustreError: 86851:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2822.831085] LustreError: 89630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787369593 with bad export cookie 6265882185690256091 [ 2822.857329] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2822.859572] LustreError: 89630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2823.439960] Lustre: server umount lustre-MDT0001 complete [ 2835.677826] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 2838.152349] Key type lgssc unregistered [ 2838.400054] LNet: 90386:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2838.410133] LNetError: 90386:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2838.436496] LNet: Removed LNI 192.168.201.151@tcp [ 2839.252364] Key type .llcrypt unregistered [ 2839.259144] Key type ._llcrypt unregistered [ 2864.500642] Key type ._llcrypt registered [ 2864.503584] Key type .llcrypt registered [ 2864.630435] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 2866.247887] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2866.332766] alg: No test for adler32 (adler32-zlib) [ 2867.439478] Lustre: Lustre: Build Version: 2.17.57_45_gb800aec [ 2867.771826] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 2869.423120] Key type lgssc registered [ 2870.613897] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2879.538339] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2879.593328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2881.182092] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2886.305575] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 2891.751342] LustreError: 91804:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2891.765561] LustreError: 91804:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2895.346511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2895.747879] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2901.006229] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 2909.491919] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2915.612226] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2922.103748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2922.434845] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2926.523250] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2929.794980] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 2938.351329] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2954.730142] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2954.744280] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2954.766397] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2956.256709] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2956.274039] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2956.278405] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2960.248158] Lustre: server umount lustre-OST0000 complete [ 2966.498082] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2966.501271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2966.518485] Lustre: Skipped 1 previous similar message [ 2966.528635] Lustre: Skipped 2 previous similar messages [ 2970.727499] Lustre: server umount lustre-MDT0000 complete [ 2971.616898] LustreError: 93643:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2971.649198] LustreError: 93643:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2974.751750] LustreError: 92538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787369745 with bad export cookie 10730975253456827334 [ 2974.766989] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2974.770034] LustreError: 92538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2975.320703] Lustre: server umount lustre-MDT0001 complete [ 2987.262346] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 2990.468547] Key type lgssc unregistered [ 2990.740793] LNet: 95362:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2990.743878] LNetError: 95362:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2990.764157] LNet: Removed LNI 192.168.201.151@tcp [ 2991.638186] Key type .llcrypt unregistered [ 2991.644720] Key type ._llcrypt unregistered [ 3016.149812] Key type ._llcrypt registered [ 3016.152288] Key type .llcrypt registered [ 3016.248215] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 3017.517803] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3017.668511] alg: No test for adler32 (adler32-zlib) [ 3018.960143] Lustre: Lustre: Build Version: 2.17.57_45_gb800aec [ 3019.352701] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 3021.104079] Key type lgssc registered [ 3022.692684] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3032.223761] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3032.251943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3047.905691] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3053.024075] LustreError: 96749:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.151@tcp: failed processing log, type 4: rc = -110 [ 3079.712395] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3085.895923] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 3093.835476] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3101.035916] Lustre: server umount lustre-OST0000 complete [ 3112.197992] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 23:38:01 (1787369881) [ 3121.329875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3121.872863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3127.286602] LustreError: 98388:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3127.327441] LustreError: 98388:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3128.884215] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 3131.365174] LustreError: 98387:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3136.481500] LustreError: 98388:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3140.355320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3140.894272] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3145.789391] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 3154.927228] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3162.633211] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3171.361115] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3171.858612] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3179.380637] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 3188.279464] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3191.359236] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3206.638139] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3206.648421] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3206.659135] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3211.751903] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3211.762152] Lustre: Skipped 2 previous similar messages [ 3212.688605] Lustre: server umount lustre-OST0000 complete [ 3216.865069] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3216.869065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3216.880577] Lustre: Skipped 2 previous similar messages [ 3216.894440] Lustre: Skipped 1 previous similar message [ 3221.984896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3222.000569] Lustre: Skipped 1 previous similar message [ 3227.104776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3227.112378] Lustre: Skipped 1 previous similar message [ 3231.199131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3231.429483] Lustre: server umount lustre-MDT0000 complete [ 3232.225533] LustreError: 98384:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3232.241855] LustreError: 98384:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3235.596745] LustreError: 98369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787370006 with bad export cookie 5555414263717629932 [ 3235.615198] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3235.619697] LustreError: 98369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3236.010956] Lustre: server umount lustre-MDT0001 complete [ 3248.954463] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 3251.608991] Key type lgssc unregistered [ 3251.918587] LNet: 102087:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3251.930591] LNetError: 102087:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3251.947673] LNet: Removed LNI 192.168.201.151@tcp [ 3252.707521] Key type .llcrypt unregistered [ 3252.713350] Key type ._llcrypt unregistered [ 3300.351805] Key type ._llcrypt registered [ 3300.353081] Key type .llcrypt registered [ 3300.503946] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 3301.423294] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3301.467534] alg: No test for adler32 (adler32-zlib) [ 3302.639378] Lustre: Lustre: Build Version: 2.17.57_45_gb800aec [ 3303.061308] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 3304.839261] Key type lgssc registered [ 3306.676104] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3316.915462] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3316.953933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3318.600760] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3324.206906] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 3329.006507] LustreError: 104274:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3329.030031] LustreError: 104274:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3333.323965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3333.639918] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3338.558475] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 3347.180269] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3354.744983] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3361.727398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3362.165577] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3364.264281] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3368.982218] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 3377.649764] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3388.895957] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3388.898281] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3388.941563] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3388.967174] Lustre: Skipped 1 previous similar message [ 3394.085780] Lustre: server umount lustre-OST0000 complete [ 3399.150415] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3399.156091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3399.163197] Lustre: Skipped 2 previous similar messages [ 3399.174888] Lustre: Skipped 1 previous similar message [ 3404.259017] LustreError: 104274:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3404.404968] Lustre: server umount lustre-MDT0000 complete [ 3407.995580] LustreError: 104253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787370179 with bad export cookie 18014504394797099613 [ 3408.002314] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3408.013721] LustreError: 104253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3408.360250] Lustre: server umount lustre-MDT0001 complete [ 3422.202261] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 3426.451343] Key type lgssc unregistered [ 3426.756064] LNet: 107811:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3426.760616] LNetError: 107811:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3427.812863] LNet: Removed LNI 192.168.201.151@tcp [ 3428.994750] Key type .llcrypt unregistered [ 3428.997555] Key type ._llcrypt unregistered [ 3453.056851] Key type ._llcrypt registered [ 3453.061193] Key type .llcrypt registered [ 3453.203929] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 3455.164282] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3455.237900] alg: No test for adler32 (adler32-zlib) [ 3456.608489] Lustre: Lustre: Build Version: 2.17.57_45_gb800aec [ 3456.906755] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 3458.719173] Key type lgssc registered [ 3460.458936] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3470.323476] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3470.364648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3485.990282] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3491.167481] LustreError: 109056:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.151@tcp: failed processing log, type 4: rc = -110 [ 3517.855428] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3525.360370] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 3534.507352] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3541.837337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3542.114449] LustreError: MGC192.168.201.151@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3542.128604] LustreError: 110082:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3542.143402] LustreError: 110082:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3542.159039] LustreError: 110082:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3542.297634] Lustre: server umount lustre-MDT0000 complete [ 3542.299303] LustreError: 110082:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3561.951402] Lustre: 108393:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787370316/real 1787370316] req@00000000e49dca8b x1874193523150208/t0(0) o400->MGC192.168.201.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787370332 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3561.995332] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3562.011835] LustreError: 109068:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3582.554488] Lustre: server umount lustre-OST0000 complete [ 3593.583037] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_hostid [ 3604.566524] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 3642.873841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3643.173210] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3643.215343] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3643.297244] Lustre: lustre-MDT0000: new disk, initializing [ 3643.383421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3643.400856] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3648.081044] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 3659.629919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3659.734225] Lustre: 112877:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3659.784457] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3659.789202] Lustre: Skipped 1 previous similar message [ 3659.880846] Lustre: lustre-MDT0001: new disk, initializing [ 3659.940967] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3659.980909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3659.997372] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3666.153141] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 3676.982467] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3683.747971] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3691.554050] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3691.871955] Lustre: lustre-OST0000: new disk, initializing [ 3691.880594] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3691.886797] Lustre: 114852:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3691.997138] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3697.242957] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3697.260102] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3697.380306] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3698.698697] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 3709.133625] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3714.687853] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3714.925326] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3720.955177] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3721.284677] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3728.362566] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3728.364336] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3728.367051] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3728.379775] LustreError: Skipped 1 previous similar message [ 3728.424909] Lustre: Skipped 1 previous similar message [ 3729.871672] Lustre: server umount lustre-OST0000 complete [ 3737.088327] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3737.114804] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3737.140344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3737.148186] Lustre: Skipped 1 previous similar message [ 3738.596168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3738.597566] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3743.712591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3743.727297] Lustre: Skipped 2 previous similar messages [ 3748.319149] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3748.615294] Lustre: server umount lustre-MDT0000 complete [ 3752.887440] LustreError: 112870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787370523 with bad export cookie 12518553779432061460 [ 3752.909441] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3753.443788] Lustre: server umount lustre-MDT0001 complete [ 3763.525778] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 23:48:53 (1787370533) [ 3767.029405] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3786.585785] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_hostid [ 3799.638537] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 3841.220521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3841.513686] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3841.550442] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3841.602247] Lustre: lustre-MDT0000: new disk, initializing [ 3841.654828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3841.671525] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3846.403751] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 3858.165407] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3858.311265] Lustre: 119753:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3858.356919] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3858.362517] Lustre: Skipped 1 previous similar message [ 3858.518808] Lustre: lustre-MDT0001: new disk, initializing [ 3858.745201] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3858.796221] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3858.810109] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3863.909583] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 3875.184659] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3882.294947] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3890.141202] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3890.494944] Lustre: lustre-OST0000: new disk, initializing [ 3890.500494] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3890.507300] Lustre: 121721:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3890.617203] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3897.542984] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 3900.470158] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3900.484604] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3900.649054] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3907.857868] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3913.319083] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3913.519592] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3918.747795] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3918.972690] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3921.388237] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3921.395449] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3921.400347] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3921.405649] Lustre: Skipped 1 previous similar message [ 3931.616702] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3931.629626] Lustre: Skipped 4 previous similar messages [ 3935.199132] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3935.487732] Lustre: server umount lustre-OST0000 complete [ 3940.834360] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3940.842291] LustreError: Skipped 1 previous similar message [ 3940.856379] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3945.984351] Lustre: server umount lustre-MDT0000 complete [ 3946.990527] LustreError: 119765:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3947.021577] LustreError: 119765:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3950.180321] LustreError: 119745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787370721 with bad export cookie 12518553779432062335 [ 3950.183139] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3950.195401] LustreError: 119745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 3950.653223] Lustre: server umount lustre-MDT0001 complete [ 3961.539539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3962.015622] LustreError: 123939:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3962.148932] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3967.346519] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 3967.470284] LustreError: 123940:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3972.586173] LustreError: 123939:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3976.678948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3982.323415] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 3991.626572] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3998.322887] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4005.843196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4014.583934] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 4024.691235] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4032.371419] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 4037.090170] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4037.124214] Lustre: Skipped 1 previous similar message [ 4037.131813] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4037.153191] Lustre: Skipped 3 previous similar messages [ 4042.777267] Lustre: server umount lustre-OST0000 complete [ 4053.423913] Lustre: server umount lustre-MDT0000 complete [ 4057.581292] LustreError: 123940:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4057.627470] LustreError: 123940:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4057.888501] LustreError: 123921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787370828 with bad export cookie 12518553779432063224 [ 4057.891822] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4057.920483] LustreError: 123921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4058.628572] Lustre: server umount lustre-MDT0001 complete [ 4070.945729] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 4074.501076] Key type lgssc unregistered [ 4074.840886] LNet: 127576:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4074.853894] LNetError: 127576:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4074.870218] LNet: Removed LNI 192.168.201.151@tcp [ 4076.000925] Key type .llcrypt unregistered [ 4076.009973] Key type ._llcrypt unregistered [ 4100.539762] Key type ._llcrypt registered [ 4100.543964] Key type .llcrypt registered [ 4100.718847] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_hostid [ 4124.254736] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 4126.297370] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4126.551139] alg: No test for adler32 (adler32-zlib) [ 4128.046607] Lustre: Lustre: Build Version: 2.17.57_45_gb800aec [ 4128.402632] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 4130.168442] Key type lgssc registered [ 4132.083553] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4173.538828] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4173.557925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4174.870617] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4174.920012] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4175.066283] Lustre: lustre-MDT0000: new disk, initializing [ 4175.249658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4175.294987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4180.088198] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 4193.283040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4193.405076] Lustre: 130737:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4193.461789] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4193.466307] Lustre: Skipped 1 previous similar message [ 4193.574408] Lustre: lustre-MDT0001: new disk, initializing [ 4193.686533] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4193.743133] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4193.764863] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4198.921288] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 4209.376422] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4216.618562] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4225.248353] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4225.768571] Lustre: lustre-OST0000: new disk, initializing [ 4225.776923] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4225.796560] Lustre: 132706:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4225.964118] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4232.310129] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4232.325538] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4232.481942] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4233.966385] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 4244.794604] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4250.680441] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4251.021428] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4255.772793] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4255.946468] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4263.405945] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4263.406324] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4263.418572] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4263.449401] Lustre: Skipped 1 previous similar message [ 4264.842374] Lustre: server umount lustre-OST0000 complete [ 4270.561599] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4270.575434] LustreError: Skipped 1 previous similar message [ 4270.591214] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4270.627369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4270.643519] Lustre: Skipped 1 previous similar message [ 4273.634037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4273.640102] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4275.899689] Lustre: server umount lustre-MDT0000 complete [ 4280.567846] LustreError: 130731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787371051 with bad export cookie 8476017213652031029 [ 4280.570638] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4280.582912] LustreError: 130731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4281.146820] Lustre: server umount lustre-MDT0001 complete [ 4291.989001] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 23:57:41 (1787371061) [ 4300.563036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4301.193173] LustreError: 135122:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4301.212611] LustreError: 135122:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4301.262535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4306.317373] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 4306.407021] LustreError: 135123:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4311.521942] LustreError: 135122:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4316.645639] LustreError: 135123:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4317.208794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4317.562926] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4323.146482] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 4331.397414] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4339.723886] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4343.201496] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4343.215746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4343.219614] Lustre: Skipped 2 previous similar messages [ 4348.403461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4348.690672] Lustre: server umount lustre-MDT0000 complete [ 4353.508124] LustreError: 136318:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4353.535797] LustreError: 136318:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4354.011386] LustreError: 135104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787371125 with bad export cookie 8476017213652031904 [ 4354.023295] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4354.032572] LustreError: 135104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4354.538257] Lustre: server umount lustre-MDT0001 complete [ 4364.902732] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 23:58:54 (1787371134) [ 4373.008530] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4388.704279] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4393.823555] LustreError: 137538:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.151@tcp: failed processing log, type 4: rc = -110 [ 4419.551413] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4426.808943] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 4435.515653] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4437.687803] Lustre: server umount lustre-OST0000 complete [ 4446.874837] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 00:00:16 (1787371216) [ 4455.498565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4455.924471] LustreError: 138987:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4455.997787] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4462.783874] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 4474.832489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4481.006240] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 4489.788247] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4497.588644] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4505.811688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4506.146164] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4506.149909] Lustre: Skipped 1 previous similar message [ 4512.393458] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 4521.910467] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4535.784864] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4535.797420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4535.818662] Lustre: Skipped 2 previous similar messages [ 4535.901046] Lustre: Skipped 3 previous similar messages [ 4541.707854] Lustre: server umount lustre-MDT0000 complete [ 4545.609413] LustreError: 138968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787371316 with bad export cookie 8476017213652032576 [ 4545.617727] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4545.634031] LustreError: 138968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4545.963604] Lustre: server umount lustre-MDT0001 complete [ 4560.516691] Lustre: server umount lustre-OST0000 complete [ 4571.324970] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 00:02:21 (1787371341) [ 4579.255497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4579.793364] LustreError: 142679:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4579.811270] LustreError: 142679:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4579.878399] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4584.782698] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 4593.618217] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4598.535159] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 4606.329043] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4613.274923] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4620.652884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4625.054963] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4627.440613] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 4635.539817] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4641.798769] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4642.114383] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4647.633364] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4647.894639] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4650.990460] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4650.997784] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4651.015818] Lustre: Skipped 1 previous similar message [ 4651.027197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4651.031941] Lustre: Skipped 3 previous similar messages [ 4654.055327] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4656.458652] Lustre: server umount lustre-OST0000 complete [ 4667.447256] Lustre: server umount lustre-MDT0000 complete [ 4669.410204] LustreError: 142674:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4669.446513] LustreError: 142674:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4672.457289] LustreError: 142660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787371443 with bad export cookie 8476017213652033647 [ 4672.464535] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4672.472551] LustreError: 142660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4672.948065] Lustre: server umount lustre-MDT0001 complete [ 4684.316532] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 00:04:13 (1787371453) [ 4694.412263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4709.983562] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4715.170810] LustreError: 146838:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.151@tcp: failed processing log, type 4: rc = -110 [ 4740.895823] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4740.914876] Lustre: Skipped 2 previous similar messages [ 4750.073954] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 4759.908409] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4768.860863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4769.303255] LustreError: 146862:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4769.491097] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4773.603373] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 4781.893195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4786.865349] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 4795.361378] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4803.069827] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4809.235954] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4809.510283] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4814.722759] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4814.998073] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4818.402476] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4818.420858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4818.449042] Lustre: Skipped 5 previous similar messages [ 4818.459073] Lustre: Skipped 5 previous similar messages [ 4823.580854] Lustre: server umount lustre-MDT0000 complete [ 4828.379926] LustreError: 146846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787371599 with bad export cookie 8476017213652034669 [ 4828.392565] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4828.397903] LustreError: 146846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4828.895870] Lustre: server umount lustre-MDT0001 complete [ 4834.372097] Lustre: server umount lustre-OST0000 complete [ 4844.217970] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 00:06:54 (1787371614) [ 4851.699255] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4867.427861] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4872.609074] LustreError: 151033:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.151@tcp: failed processing log, type 4: rc = -110 [ 4905.620391] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 4913.552355] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4921.146205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4921.661929] LustreError: 151058:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4921.701267] LustreError: 151058:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 4921.898253] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4927.198737] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 4936.476984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4941.350335] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 4950.952165] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4956.915129] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4964.428106] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4964.671732] Lustre: lustre-OST0001: new disk, initializing [ 4964.683708] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4964.689670] Lustre: 153905:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4970.577337] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4970.586901] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4970.674592] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4971.556093] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 4983.156801] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4988.743685] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4989.012856] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4993.858242] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4993.986729] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4996.583529] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4996.587608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4996.618527] Lustre: Skipped 2 previous similar messages [ 4996.638689] Lustre: Skipped 2 previous similar messages [ 5001.880313] Lustre: server umount lustre-OST0000 complete [ 5011.946829] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5011.958350] LustreError: Skipped 1 previous similar message [ 5021.152283] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5021.506820] Lustre: server umount lustre-OST0001 complete [ 5039.072095] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5039.313811] Lustre: server umount lustre-MDT0000 complete [ 5042.646074] LustreError: 151041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787371813 with bad export cookie 8476017213652035621 [ 5042.647123] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5042.657105] LustreError: 151041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5042.962275] Lustre: server umount lustre-MDT0001 complete [ 5064.892823] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 00:10:35 (1787371835) [ 5066.517606] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 5067.826395] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 5069.530578] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 00:10:39 (1787371839) [ 5075.398864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5075.628127] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5075.690832] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5075.954144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5075.960293] Lustre: Skipped 6 previous similar messages [ 5081.275800] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 5090.698748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5090.882505] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5090.988306] Lustre: 157467:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5096.162161] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 5106.019864] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5115.986754] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5124.947766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5125.506166] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5132.477782] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 5133.838573] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 5142.967986] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5148.233609] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5148.488479] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5152.930528] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5153.140394] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5157.858359] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5157.866093] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5157.882095] Lustre: Skipped 4 previous similar messages [ 5157.891093] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5157.897171] Lustre: Skipped 12 previous similar messages [ 5169.119170] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5169.318400] Lustre: server umount lustre-OST0000 complete [ 5192.931548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5198.444694] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5198.469495] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5198.471565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5199.890802] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 5207.923233] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5214.526654] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5214.869761] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5221.011077] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5221.373681] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5228.360929] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff922546f36800.ost_server_uuid 50 [ 5230.527757] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff922546f36800.ost_server_uuid in FULL state after 0 sec [ 5230.838409] LustreError: 157477:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.201.51@tcp) returned error from blocking AST (req@00000000a09436df x1874194228067840 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 000000007c0b24a6/0x75a0dcbf79d3717f lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.51@tcp remote: 0xd2a6bd21ddec4321 expref: 5 pid: 157476 timeout: 5330 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5230.872261] LustreError: lustre-MDT0000: A client on nid 192.168.201.51@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5230.878591] LustreError: 157465:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.201.51@tcp ns: mdt-lustre-MDT0000_UUID lock: 000000007c0b24a6/0x75a0dcbf79d3717f lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.51@tcp remote: 0xd2a6bd21ddec4321 expref: 6 pid: 157476 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5236.708411] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5236.716625] LustreError: Skipped 1 previous similar message [ 5250.018079] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5250.154678] Lustre: server umount lustre-OST0000 complete [ 5261.428694] Lustre: server umount lustre-MDT0000 complete [ 5265.378116] LustreError: 157481:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5265.403749] LustreError: 157481:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 5265.632095] LustreError: 157461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787372036 with bad export cookie 8476017213652036804 [ 5265.634354] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5265.640496] LustreError: 157461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5272.141394] Lustre: server umount lustre-MDT0001 complete [ 5284.758636] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 5287.415596] Key type lgssc unregistered [ 5287.704808] LNet: 163589:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5287.714728] LNetError: 163589:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5287.741688] LNet: Removed LNI 192.168.201.151@tcp [ 5288.468272] Key type .llcrypt unregistered [ 5288.470241] Key type ._llcrypt unregistered [ 5317.639623] Key type ._llcrypt registered [ 5317.644522] Key type .llcrypt registered [ 5317.763903] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 5319.053031] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5319.091169] alg: No test for adler32 (adler32-zlib) [ 5320.165868] Lustre: Lustre: Build Version: 2.17.57_45_gb800aec [ 5320.442359] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 5322.183363] Key type lgssc registered [ 5323.448558] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5332.053665] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5332.096533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5333.479949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5338.341351] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 5343.720850] LustreError: 165005:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5343.753227] LustreError: 165005:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5348.238327] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5348.535764] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5353.914766] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 5362.123694] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5368.766864] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5376.023757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5376.559758] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5378.631972] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5383.871134] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 5391.772956] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5397.449547] Lustre: Failing over lustre-MDT0000 [ 5397.679577] Lustre: server umount lustre-MDT0000 complete [ 5398.499694] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5398.510691] LustreError: 165005:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5398.530615] LustreError: 165005:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5403.619216] LustreError: 165000:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5403.638155] LustreError: 165000:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5408.742333] LustreError: 165006:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5408.773492] LustreError: 165006:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5409.660319] LustreError: 164985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.51@tcp arrived at 1787372180 with bad export cookie 4788110843340160144 [ 5413.871112] LustreError: 165005:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5413.901282] LustreError: 165005:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5414.893203] Lustre: 164316:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787372169/real 1787372169] req@00000000d3a41b01 x1874195477718528/t0(0) o400->MGC192.168.201.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787372185 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5414.921043] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5418.326619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5423.008723] LustreError: 165734:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5423.034805] LustreError: 165734:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 5424.468978] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5424.591319] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5429.750177] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5429.759387] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5429.976279] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 5432.493172] Lustre: lustre-MDT0000: Denying connection for new client 5437f52c-0774-4f85-9a24-bd43933907c3 (at 192.168.201.51@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:07 [ 5444.579598] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5444.586241] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5444.601634] Lustre: Skipped 1 previous similar message [ 5444.605983] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5447.504582] Lustre: server umount lustre-OST0000 complete [ 5450.894025] LustreError: 168734:0:(ldlm_lib.c:3029:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5450.899552] Lustre: 168083:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5450.905260] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5450.916301] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5450.922738] LustreError: 168083:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000fe6800a1 x1874195477739776/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5450.931480] LustreError: 168083:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5450.936765] LustreError: 168083:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5450.941850] Lustre: lustre-MDT0000: Recovery over after 0:21, of 2 clients 0 recovered and 2 were evicted. [ 5451.154302] Lustre: server umount lustre-MDT0000 complete [ 5454.926384] LustreError: 164987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787372225 with bad export cookie 4788110843340160249 [ 5454.934479] LustreError: 164313:0:(import.c:707:ptlrpc_connect_import_locked()) can't connect to a closed import [ 5454.937656] LustreError: 164987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 5454.938753] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5457.243287] Lustre: server umount lustre-MDT0001 complete [ 5469.179076] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 5471.762833] Key type lgssc unregistered [ 5472.041494] LNet: 169490:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5472.049971] LNetError: 169490:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5472.068711] LNet: Removed LNI 192.168.201.151@tcp [ 5472.918123] Key type .llcrypt unregistered [ 5472.920540] Key type ._llcrypt unregistered [ 5500.406825] Key type ._llcrypt registered [ 5500.411096] Key type .llcrypt registered [ 5500.554693] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 5502.007566] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5502.188569] alg: No test for adler32 (adler32-zlib) [ 5503.418977] Lustre: Lustre: Build Version: 2.17.57_45_gb800aec [ 5503.737505] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 5505.455295] Key type lgssc registered [ 5506.565702] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5514.400179] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5514.413366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5515.941409] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5520.475210] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 5525.480071] LustreError: 170913:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5525.508103] LustreError: 170913:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5529.114641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5529.710753] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5535.009877] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 5542.991699] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5549.037191] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5556.805308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5557.237656] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5560.321062] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5563.660474] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 5571.990401] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5579.747836] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5579.752878] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5579.759690] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5579.811995] Lustre: Skipped 1 previous similar message [ 5584.879551] LustreError: 172736:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5584.906030] LustreError: 172736:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5584.966274] Lustre: server umount lustre-OST0000 complete [ 5589.991023] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5590.006454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5590.016318] Lustre: Skipped 2 previous similar messages [ 5595.107611] LustreError: 170914:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5595.145062] LustreError: 170914:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5596.710226] Lustre: server umount lustre-MDT0000 complete [ 5600.230766] LustreError: 171642:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5600.250943] LustreError: 171642:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5601.165216] LustreError: 170894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787372372 with bad export cookie 4463059397461796487 [ 5601.172367] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5601.185773] LustreError: 170894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5601.663523] Lustre: server umount lustre-MDT0001 complete [ 5614.144748] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 5617.363597] Key type lgssc unregistered [ 5617.731241] LNet: 174449:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5617.746680] LNetError: 174449:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5617.762490] LNet: Removed LNI 192.168.201.151@tcp [ 5618.812605] Key type .llcrypt unregistered [ 5618.815235] Key type ._llcrypt unregistered [ 5651.395857] Key type ._llcrypt registered [ 5651.415488] Key type .llcrypt registered [ 5651.512337] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 5652.544215] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5652.735183] alg: No test for adler32 (adler32-zlib) [ 5653.852379] Lustre: Lustre: Build Version: 2.17.57_45_gb800aec [ 5654.020724] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 5655.647251] Key type lgssc registered [ 5656.715231] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5663.809477] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5663.823926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5665.111675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5669.151859] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 5675.491478] LustreError: 176153:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5675.508434] LustreError: 176153:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5677.952981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5678.339582] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5682.398962] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 5690.369760] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5696.365824] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5703.584430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5703.943235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5709.614305] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 5710.141809] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5717.489866] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5728.737082] LustreError: lustre-OST0000-osc-MDT0001: operation ost_create to node 0@lo failed: rc = -107 [ 5728.745621] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5728.748314] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5728.754099] LustreError: Skipped 1 previous similar message [ 5728.788220] Lustre: Skipped 1 previous similar message [ 5733.864373] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5733.877881] Lustre: Skipped 1 previous similar message [ 5734.953556] Lustre: server umount lustre-OST0000 complete [ 5738.976831] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5738.985375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5738.991360] Lustre: Skipped 1 previous similar message [ 5739.019371] Lustre: Skipped 1 previous similar message [ 5744.096223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5744.941680] Lustre: server umount lustre-MDT0000 complete [ 5749.221622] LustreError: 176154:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5749.261736] LustreError: 176154:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5749.489798] LustreError: 176135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787372520 with bad export cookie 12920559762417314611 [ 5749.498775] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5749.504701] LustreError: 176135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5749.508609] LustreError: 176883:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0001: can't precreate: rc = -5 [ 5749.508675] LustreError: 176883:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0001: cannot precreate objects: rc = -5 [ 5749.976140] Lustre: server umount lustre-MDT0001 complete [ 5763.948710] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 5768.105447] Key type lgssc unregistered [ 5768.375647] LNet: 179690:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5768.388833] LNetError: 179690:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5768.407712] LNet: Removed LNI 192.168.201.151@tcp [ 5769.477529] Key type .llcrypt unregistered [ 5769.480918] Key type ._llcrypt unregistered [ 5791.575613] Key type ._llcrypt registered [ 5791.578539] Key type .llcrypt registered [ 5791.724836] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 5793.539097] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5793.579243] alg: No test for adler32 (adler32-zlib) [ 5794.791714] Lustre: Lustre: Build Version: 2.17.57_45_gb800aec [ 5795.244122] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 5796.999633] Key type lgssc registered [ 5798.205642] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5809.588269] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5809.609328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5810.931160] Lustre: *** cfs_fail_loc=135, val=0*** [ 5811.019886] LustreError: 181208:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5811.035857] LustreError: 181208:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.151@tcp: cfg command failed: rc = -2 [ 5811.052216] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5811.052216] [ 5811.069648] LustreError: MGC192.168.201.151@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5811.083892] LustreError: 181169:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5811.090712] LustreError: 181169:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5811.096581] LustreError: 181169:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 5811.105256] LustreError: 181169:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5811.178796] Lustre: server umount lustre-MDT0000 complete [ 5811.180879] LustreError: 181169:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5818.253333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5818.692160] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5824.954178] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 5828.065751] LustreError: 181579:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5828.083173] LustreError: 181579:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5833.209184] LustreError: 181580:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5834.172035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5834.530489] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5838.597414] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 5845.616351] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5852.201557] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5855.200611] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5855.246602] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5855.253440] Lustre: Skipped 1 previous similar message [ 5859.362601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5859.377649] Lustre: Skipped 1 previous similar message [ 5860.750484] Lustre: server umount lustre-MDT0000 complete [ 5864.422781] LustreError: 181576:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5864.450234] LustreError: 181576:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5865.070662] LustreError: 181561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787372636 with bad export cookie 1875967607675721785 [ 5865.073701] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5865.085075] LustreError: 181561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5865.439462] Lustre: server umount lustre-MDT0001 complete [ 5877.076222] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 5880.448882] Key type lgssc unregistered [ 5880.803970] LNet: 183906:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5880.820372] LNetError: 183906:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5881.829918] LNet: Removed LNI 192.168.201.151@tcp [ 5882.762642] Key type .llcrypt unregistered [ 5882.765171] Key type ._llcrypt unregistered