[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 772299394 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002407] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004017] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008004] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009014] pid_max: default: 32768 minimum: 301 [ 0.010132] LSM: Security Framework initializing [ 0.011080] Yama: becoming mindful. [ 0.012042] SELinux: Initializing. [ 0.013081] *** VALIDATE selinux *** [ 0.021772] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026810] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027151] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028113] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029124] *** VALIDATE tmpfs *** [ 0.030494] *** VALIDATE proc *** [ 0.031250] *** VALIDATE cgroup *** [ 0.033004] *** VALIDATE cgroup2 *** [ 0.034288] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036100] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037080] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038030] Spectre V2 : User space: Vulnerable [ 0.039009] Speculative Store Bypass: Vulnerable [ 0.042365] debug: unmapping init [mem 0xffffffff93859000-0xffffffff93860fff] [ 0.046000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046787] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047036] ... version: 2 [ 0.048015] ... bit width: 48 [ 0.049014] ... generic registers: 4 [ 0.050076] ... value mask: 0000ffffffffffff [ 0.051018] ... max period: 00007fffffffffff [ 0.052026] ... fixed-purpose events: 3 [ 0.053015] ... event mask: 000000070000000f [ 0.054426] rcu: Hierarchical SRCU implementation. [ 0.056923] smp: Bringing up secondary CPUs ... [ 0.058381] x86: Booting SMP configuration: [ 0.059094] .... node #0, CPUs: #1 #2 #3 [ 0.070073] smp: Brought up 1 node, 4 CPUs [ 0.072014] smpboot: Max logical packages: 1 [ 0.073013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.164031] node 0 deferred pages initialised in 89ms [ 0.187199] devtmpfs: initialized [ 0.190111] x86/mm: Memory block size: 128MB [ 0.199244] gcov: version magic: 0x41383552 [ 0.202486] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.203093] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.204377] pinctrl core: initialized pinctrl subsystem [ 0.205192] [ 0.205795] ************************************************************* [ 0.206019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.207017] ** ** [ 0.208017] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.209016] ** ** [ 0.210032] ** This means that this kernel is built to expose internal ** [ 0.211016] ** IOMMU data structures, which may compromise security on ** [ 0.212016] ** your system. ** [ 0.213017] ** ** [ 0.214016] ** If you see this message and you are not debugging the ** [ 0.215020] ** kernel, report this immediately to your vendor! ** [ 0.216016] ** ** [ 0.217017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.218015] ************************************************************* [ 0.220675] NET: Registered protocol family 16 [ 0.223636] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.227074] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.230062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.234150] cpuidle: using governor menu [ 0.235918] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.239699] PCI: Using configuration type 1 for base access [ 0.242145] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.256095] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.259035] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.262366] cryptd: max_cpu_qlen set to 1000 [ 0.266272] ACPI: Added _OSI(Module Device) [ 0.269106] ACPI: Added _OSI(Processor Device) [ 0.271132] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.274022] ACPI: Added _OSI(Processor Aggregator Device) [ 0.279647] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.293960] ACPI: Interpreter enabled [ 0.295091] ACPI: PM: (supports S0 S3 S4 S5) [ 0.300025] ACPI: Using IOAPIC for interrupt routing [ 0.302294] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.308512] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.321177] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.324067] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.327027] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.330216] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.336957] acpiphp: Slot [2] registered [ 0.339225] acpiphp: Slot [5] registered [ 0.340271] acpiphp: Slot [6] registered [ 0.342324] acpiphp: Slot [7] registered [ 0.344212] acpiphp: Slot [8] registered [ 0.346196] acpiphp: Slot [9] registered [ 0.348197] acpiphp: Slot [10] registered [ 0.350219] acpiphp: Slot [3] registered [ 0.352171] acpiphp: Slot [4] registered [ 0.354154] acpiphp: Slot [11] registered [ 0.356226] acpiphp: Slot [12] registered [ 0.358176] acpiphp: Slot [13] registered [ 0.360160] acpiphp: Slot [14] registered [ 0.362159] acpiphp: Slot [15] registered [ 0.364191] acpiphp: Slot [16] registered [ 0.366227] acpiphp: Slot [17] registered [ 0.368233] acpiphp: Slot [18] registered [ 0.370259] acpiphp: Slot [19] registered [ 0.372179] acpiphp: Slot [20] registered [ 0.374159] acpiphp: Slot [21] registered [ 0.376160] acpiphp: Slot [22] registered [ 0.378232] acpiphp: Slot [23] registered [ 0.381230] acpiphp: Slot [24] registered [ 0.383248] acpiphp: Slot [25] registered [ 0.384299] acpiphp: Slot [26] registered [ 0.386137] acpiphp: Slot [27] registered [ 0.388155] acpiphp: Slot [28] registered [ 0.390166] acpiphp: Slot [29] registered [ 0.391192] acpiphp: Slot [30] registered [ 0.393200] acpiphp: Slot [31] registered [ 0.395085] PCI host bridge to bus 0000:00 [ 0.397034] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.400035] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.403063] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.406032] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.409035] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.413050] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.416298] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.420000] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.423754] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.434980] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.440060] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.443026] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.446026] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.450029] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.452839] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.456920] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.460063] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.464216] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.470021] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.485021] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.493077] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.499730] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.508023] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.516030] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.542020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.553457] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.607045] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.622022] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.640069] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.651432] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.659021] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.676229] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.702019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.733249] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.751024] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.760022] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.782025] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.795513] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.806022] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.813023] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.827029] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.837192] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.838000] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.838000] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.839000] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.859119] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.860639] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.863389] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.865497] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.866230] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.871140] iommu: Default domain type: Passthrough [ 0.872000] SCSI subsystem initialized [ 0.873235] ACPI: bus type USB registered [ 0.877179] usbcore: registered new interface driver usbfs [ 0.880376] usbcore: registered new interface driver hub [ 0.883127] usbcore: registered new device driver usb [ 0.886728] pps_core: LinuxPPS API ver. 1 registered [ 0.890016] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.896091] PTP clock support registered [ 0.900959] EDAC MC: Ver: 3.0.0 [ 0.903274] PCI: Using ACPI for IRQ routing [ 0.908069] NetLabel: Initializing [ 0.912077] NetLabel: domain hash size = 128 [ 0.914017] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.918105] NetLabel: unlabeled traffic allowed by default [ 0.921532] vgaarb: loaded [ 0.923569] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.927017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.934198] clocksource: Switched to clocksource kvm-clock [ 1.088162] VFS: Disk quotas dquot_6.6.0 [ 1.090181] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.093294] *** VALIDATE ramfs *** [ 1.094772] *** VALIDATE hugetlbfs *** [ 1.096745] pnp: PnP ACPI init [ 1.099660] pnp: PnP ACPI: found 6 devices [ 1.153649] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.158303] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.163328] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.166282] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.170361] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.177556] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.190465] NET: Registered protocol family 2 [ 1.211513] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.231608] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.244884] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.252701] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.257368] TCP: Hash tables configured (established 65536 bind 65536) [ 1.261660] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.266402] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.270688] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.282624] NET: Registered protocol family 1 [ 1.289767] RPC: Registered named UNIX socket transport module. [ 1.295437] RPC: Registered udp transport module. [ 1.297586] RPC: Registered tcp transport module. [ 1.299700] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.302695] NET: Registered protocol family 44 [ 1.304712] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.307596] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.309718] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.312612] PCI: CLS 0 bytes, default 64 [ 1.314725] Unpacking initramfs... [ 3.262644] debug: unmapping init [mem 0xffff9e1a3cc54000-0xffff9e1a3ffbffff] [ 3.270048] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.272714] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.277709] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 4.980979] Initialise system trusted keyrings [ 4.987432] Key type blacklist registered [ 5.017630] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 5.067339] zbud: loaded [ 5.127469] *** VALIDATE nfs *** [ 5.134268] *** VALIDATE nfs4 *** [ 5.138316] pstore: using deflate compression [ 5.152175] Platform Keyring initialized [ 5.584460] NET: Registered protocol family 38 [ 5.586865] Key type asymmetric registered [ 5.593965] Asymmetric key parser 'x509' registered [ 5.604753] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 5.609831] io scheduler mq-deadline registered [ 5.613852] io scheduler kyber registered [ 5.621921] io scheduler bfq registered [ 5.631893] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 5.646448] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 5.659613] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 5.666662] ACPI: Power Button [PWRF] [ 5.687209] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 5.698226] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 5.715177] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 5.722517] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 5.785855] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 5.827147] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 5.857867] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 5.863470] Non-volatile memory driver v1.3 [ 5.865382] Linux agpgart interface v0.103 [ 5.900178] virtio_blk virtio1: [vda] 145896 512-byte logical blocks (74.7 MB/71.2 MiB) [ 5.903564] vda: detected capacity change from 0 to 74698752 [ 6.007337] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 6.027283] vdb: detected capacity change from 0 to 1073741824 [ 6.116393] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 6.153469] vdc: detected capacity change from 0 to 2621440000 [ 6.212672] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 6.216039] vdd: detected capacity change from 0 to 2621440000 [ 6.233407] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 6.237192] vde: detected capacity change from 0 to 4294967296 [ 6.261611] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 6.265602] vdf: detected capacity change from 0 to 4294967296 [ 6.276940] libphy: Fixed MDIO Bus: probed [ 6.288404] usbcore: registered new interface driver usbserial_generic [ 6.291866] usbserial: USB Serial support registered for generic [ 6.294893] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 6.302361] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 6.305701] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 6.308756] mousedev: PS/2 mouse device common for all mice [ 6.313125] rtc_cmos 00:05: RTC can wake from S4 [ 6.318991] rtc_cmos 00:05: registered as rtc0 [ 6.321871] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 6.324617] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 6.333662] intel_pstate: CPU model not supported [ 6.337681] hid: raw HID events driver (C) Jiri Kosina [ 6.340615] usbcore: registered new interface driver usbhid [ 6.341447] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 6.346044] usbhid: USB HID core driver [ 6.346302] drop_monitor: Initializing network drop monitor service [ 6.346445] Initializing XFRM netlink socket [ 6.347946] NET: Registered protocol family 10 [ 6.364556] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 6.367221] Segment Routing with IPv6 [ 6.373708] NET: Registered protocol family 17 [ 6.376193] mpls_gso: MPLS GSO support [ 6.385521] RAS: Correctable Errors collector initialized. [ 6.402830] AVX version of gcm_enc/dec engaged. [ 6.405370] AES CTR mode by8 optimization enabled [ 6.517626] sched_clock: Marking stable (6517602972, 0)->(8175070110, -1657467138) [ 6.521855] registered taskstats version 1 [ 6.524389] Loading compiled-in X.509 certificates [ 6.527208] zswap: loaded using pool lzo/zbud [ 6.561686] Key type big_key registered [ 6.578476] Key type encrypted registered [ 6.584521] ima: No TPM chip found, activating TPM-bypass! [ 6.591214] ima: Allocated hash algorithm: sha1 [ 6.593779] ima: No architecture policies found [ 6.595513] evm: Initialising EVM extended attributes: [ 6.598406] evm: security.selinux [ 6.600365] evm: security.ima [ 6.601961] evm: security.capability [ 6.603647] evm: HMAC attrs: 0x1 [ 6.606901] rtc_cmos 00:05: setting system clock to 2026-08-08 12:07:30 UTC (1786190850) [ 6.615233] debug: unmapping init [mem 0xffffffff94803000-0xffffffff949fffff] [ 6.618757] debug: unmapping init [mem 0xffffffff93582000-0xffffffff93858fff] [ 6.631286] Write protecting the kernel read-only data: 28672k [ 6.636155] debug: unmapping init [mem 0xffffffff91c03000-0xffffffff91dfffff] [ 6.640963] debug: unmapping init [mem 0xffffffff92514000-0xffffffff925fffff] [ 6.746504] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 6.771765] systemd[1]: Detected virtualization kvm. [ 6.776621] systemd[1]: Detected architecture x86-64. [ 6.781616] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 6.819690] systemd[1]: No hostname configured. [ 6.821932] systemd[1]: Set hostname to . [ 6.824524] random: systemd: uninitialized urandom read (16 bytes read) [ 6.827626] systemd[1]: Initializing machine ID from random generator. [ 6.972282] random: ln: uninitialized urandom read (6 bytes read) [ 7.154731] random: systemd: uninitialized urandom read (16 bytes read) [ 7.159838] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 7.166258] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 7.178863] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. [ OK ] Reached target Sockets. Starting Journal Service... Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 8.311364] device-mapper: uevent: version 1.0.3 [ 8.315732] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 10.709234] random: fast init done [ 10.876257] virtio_net virtio0 ens2: renamed from eth0 [ 11.284279] scsi host0: ata_piix [ 11.323191] scsi host1: ata_piix [ 11.326577] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 11.359349] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 18.277391] random: crng init done [ 18.290571] random: 7 urandom warning(s) missed due to ratelimiting [ 21.045811] dracut-initqueue[587]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 22.879269] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 26.151329] printk: systemd: 26 output lines suppressed due to ratelimiting [ 26.836211] SELinux: Disabled at runtime. [ 27.053496] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 27.066296] systemd[1]: Detected virtualization kvm. [ 27.068904] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 29.670250] systemd[1]: initrd-switch-root.service: Succeeded. [ 29.679945] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 29.694323] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 29.706276] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 29.711760] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 29.724215] systemd[1]: Starting Journal Service... Starting Journal Service... [ 29.739426] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting POSIX Message Queue File System... [ OK ] Stopped target Switch Root. [ 29.871473] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Huge Pages File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on udev Control Socket. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Kernel Debug File System... Starting Apply Kernel Variables... [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-getty.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-serial\x2dgetty.slice. Starting udev Coldplug all Devices... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 33.298688] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 35.149430] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 35.708318] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 37.146037] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 37.369281] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…only root support (10s / no limit) [** ] A start job is running for Configur…only root support (10s / no limit) [*** ] A start job is running for Configur…only root support (11s / no limit) [ *** ] A start job is running for Configur…only root support (11s / no limit) [ *** ] A start job is running for Configur…only root support (12s / no limit)[ 42.353842] Key type dns_resolver registered [ ***] A start job is running for Configur…only root support (12s / no limit) [ **] A start job is running for Configur…only root support (13s / n[ 43.013220] NFS: Registering the id_resolver key type o limit)[ 43.020606] Key type id_resolver registered [ 43.028787] Key type id_legacy registered [ *] A start job is running for Configur…only root support (13s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. [ 55.272367] hrtimer: interrupt took 2799749 ns Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg334-server login: [ 102.821369] libcfs: loading out-of-tree module taints kernel. [ 102.937451] Key type ._llcrypt registered [ 102.939863] Key type .llcrypt registered [ 103.027692] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 121.160936] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 123.986538] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 123.995193] alg: No test for adler32 (adler32-zlib) [ 125.341357] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 126.138455] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 127.920487] Key type lgssc registered [ 129.826772] Lustre: Echo OBD driver; http://www.lustre.org/ [ 147.167390] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 185.606186] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 198.169672] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 198.195260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 199.444968] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 199.470163] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 199.607792] Lustre: lustre-MDT0000: new disk, initializing [ 199.698894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 199.726914] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 204.225980] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 218.413185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 218.505779] Lustre: 6506:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 218.529622] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 218.533557] Lustre: Skipped 1 previous similar message [ 218.593531] Lustre: lustre-MDT0001: new disk, initializing [ 218.644312] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 218.743137] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 218.763124] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 223.064957] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 227.731571] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 236.349475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 236.675829] Lustre: lustre-OST0000: new disk, initializing [ 236.678827] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 236.683692] Lustre: 8441:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 236.761243] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 241.809244] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 241.846949] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 242.086442] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 243.471362] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 256.751215] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 256.874757] Lustre: lustre-OST0001: new disk, initializing [ 256.879548] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 256.891375] Lustre: 9514:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 256.987069] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 262.158472] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 262.164829] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 262.247114] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 263.643192] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 274.524888] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 283.988468] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 289.842327] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing check_logdir /tmp/testlogs/ [ 294.076813] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing yml_node [ 300.435624] Lustre: DEBUG MARKER: Client: 2.17.56.51 [ 302.831995] Lustre: DEBUG MARKER: MDS: 2.17.56.51 [ 305.113300] Lustre: DEBUG MARKER: OSS: 2.17.56.51 [ 306.874570] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Aug 8 08:12:29 EDT 2026 [ 322.249843] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 323.466846] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 338.917130] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 338.919988] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 338.947361] Lustre: Skipped 1 previous similar message [ 338.976237] Lustre: Skipped 3 previous similar messages [ 343.942906] Lustre: server umount lustre-MDT0000 complete [ 347.370782] LustreError: 6499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786191191 with bad export cookie 1261973834121984028 [ 347.381429] LustreError: 6499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 347.382888] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 349.154770] LustreError: 10313:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 349.156663] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 349.171701] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 349.181113] LustreError: 10313:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 349.208089] Lustre: Skipped 3 previous similar messages [ 352.167487] LustreError: 6512:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 352.168179] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 352.179216] LustreError: 6512:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 352.232483] Lustre: Skipped 2 previous similar messages [ 353.985951] Lustre: server umount lustre-MDT0001 complete [ 358.218480] Lustre: server umount lustre-OST0000 complete [ 361.238665] Lustre: server umount lustre-OST0001 complete [ 367.288489] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 376.616634] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 410.428434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 410.776119] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 410.845781] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 410.993159] Lustre: lustre-MDT0000: new disk, initializing [ 411.056152] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 411.071523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 416.234776] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 425.766975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 425.864767] Lustre: 15072:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 425.915765] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 425.923880] Lustre: Skipped 1 previous similar message [ 425.988907] Lustre: lustre-MDT0001: new disk, initializing [ 426.075731] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 426.205355] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 426.211217] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 430.855714] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 439.414537] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 445.629225] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 451.219853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 451.486342] Lustre: lustre-OST0000: new disk, initializing [ 451.490606] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 451.496551] Lustre: 17039:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 451.586982] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 453.492529] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 453.506363] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 453.695397] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 457.140314] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 466.076803] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 470.650397] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 471.123928] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 474.910964] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 475.110417] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 479.209495] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 479.212986] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 479.221500] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 479.236296] Lustre: Skipped 1 previous similar message [ 479.248493] Lustre: Skipped 1 previous similar message [ 482.908934] Lustre: server umount lustre-OST0000 complete [ 487.395071] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 487.399528] LustreError: Skipped 1 previous similar message [ 487.402077] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 487.410390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 492.267435] Lustre: server umount lustre-MDT0000 complete [ 494.573320] LustreError: 15084:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 494.683358] LustreError: 15084:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 495.366783] LustreError: 15063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786191339 with bad export cookie 1261973834121985561 [ 495.368021] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 495.385581] LustreError: 15063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 495.615155] Lustre: server umount lustre-MDT0001 complete [ 499.059892] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 08:15:41 (1786191341) [ 504.082915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 504.389218] LustreError: 19306:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 504.451194] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 508.419883] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 509.923439] LustreError: 19307:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 515.069584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 519.296656] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 525.381321] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 530.124196] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 535.716094] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 541.364613] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 546.696368] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 559.600075] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 559.601924] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 559.603546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 559.603553] Lustre: Skipped 2 previous similar messages [ 559.624898] Lustre: Skipped 2 previous similar messages [ 564.717726] Lustre: server umount lustre-OST0000 complete [ 569.828866] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 573.540923] Lustre: server umount lustre-MDT0000 complete [ 574.949246] LustreError: 20595:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 574.972622] LustreError: 20595:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 576.355483] LustreError: 19287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786191420 with bad export cookie 1261973834121986443 [ 576.363642] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 576.364835] LustreError: 19287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 576.633857] Lustre: server umount lustre-MDT0001 complete [ 584.966173] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 587.041277] Key type lgssc unregistered [ 587.260560] LNet: 22854:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 587.268763] LNetError: 22854:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 587.278404] LNet: Removed LNI 192.168.203.134@tcp [ 588.046256] Key type .llcrypt unregistered [ 588.052237] Key type ._llcrypt unregistered [ 607.004670] Key type ._llcrypt registered [ 607.007304] Key type .llcrypt registered [ 607.100170] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 607.886598] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 607.901905] alg: No test for adler32 (adler32-zlib) [ 608.903742] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 609.183081] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 610.896176] Key type lgssc registered [ 611.731519] Lustre: Echo OBD driver; http://www.lustre.org/ [ 617.271981] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 617.290873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 618.550805] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 621.382886] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 627.674406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 628.125415] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 631.156695] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 636.977429] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 640.928992] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 645.756701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 645.994487] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 650.775581] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 656.024242] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 664.429039] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 672.737355] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 672.741641] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 672.750343] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 674.785307] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 674.787482] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 674.789122] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 677.430698] Lustre: server umount lustre-OST0000 complete [ 684.513682] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 684.518179] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 684.528174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 684.532299] Lustre: Skipped 1 previous similar message [ 686.321555] Lustre: server umount lustre-MDT0000 complete [ 688.886303] LustreError: 24226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786191532 with bad export cookie 7654456847419001761 [ 688.901068] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 689.091444] Lustre: server umount lustre-MDT0001 complete [ 696.542285] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 698.679887] Key type lgssc unregistered [ 698.897361] LNet: 28223:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 698.902889] LNetError: 28223:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 698.918129] LNet: Removed LNI 192.168.203.134@tcp [ 699.616313] Key type .llcrypt unregistered [ 699.618058] Key type ._llcrypt unregistered [ 717.175558] Key type ._llcrypt registered [ 717.178752] Key type .llcrypt registered [ 717.245583] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 717.938531] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 717.959536] alg: No test for adler32 (adler32-zlib) [ 718.917809] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 719.061785] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 720.704158] Key type lgssc registered [ 721.445818] Lustre: Echo OBD driver; http://www.lustre.org/ [ 726.273549] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 726.288267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 727.551759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 730.127910] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 736.432917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 736.728814] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 740.179677] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 745.221792] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 749.443836] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 758.882822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 759.098032] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 762.837536] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 767.457405] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 772.471325] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 776.674458] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 776.680114] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 776.689954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 777.701363] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 777.702860] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 777.704969] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 777.721842] Lustre: Skipped 1 previous similar message [ 778.541499] Lustre: server umount lustre-OST0000 complete [ 782.819877] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 782.820193] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 782.831217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 782.841244] Lustre: Skipped 1 previous similar message [ 787.056611] Lustre: server umount lustre-MDT0000 complete [ 787.938866] LustreError: 31342:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 787.946859] LustreError: 31342:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 789.696430] LustreError: 29595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786191633 with bad export cookie 5065155237167057142 [ 789.700178] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 789.704667] LustreError: 29595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 789.894607] Lustre: server umount lustre-MDT0001 complete [ 797.406157] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 799.306731] Key type lgssc unregistered [ 799.492821] LNet: 33492:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 799.508229] LNetError: 33492:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 800.552371] LNet: Removed LNI 192.168.203.134@tcp [ 801.127825] Key type .llcrypt unregistered [ 801.132362] Key type ._llcrypt unregistered [ 816.561091] Key type ._llcrypt registered [ 816.562897] Key type .llcrypt registered [ 816.646721] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 817.154084] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 817.227161] alg: No test for adler32 (adler32-zlib) [ 818.175996] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 818.366113] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 820.000198] Key type lgssc registered [ 820.835305] Lustre: Echo OBD driver; http://www.lustre.org/ [ 825.977274] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 825.988097] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 827.264824] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 830.282705] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 837.604316] LustreError: 34883:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 837.618938] LustreError: 34883:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 837.710203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 837.955599] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 840.645407] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 845.156661] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 848.364440] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 852.011297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 852.228751] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 853.300743] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 855.724890] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 860.151312] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 872.929542] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 872.941054] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 872.941956] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 872.961819] Lustre: Skipped 1 previous similar message [ 875.203370] Lustre: server umount lustre-OST0000 complete [ 878.049110] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 878.051927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 878.057613] Lustre: Skipped 1 previous similar message [ 878.065138] Lustre: Skipped 2 previous similar messages [ 883.169233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 883.173732] Lustre: Skipped 1 previous similar message [ 883.402336] Lustre: server umount lustre-MDT0000 complete [ 885.451982] LustreError: 34861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786191729 with bad export cookie 7053369063978726286 [ 885.454294] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 885.463037] LustreError: 34861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 885.657137] Lustre: server umount lustre-MDT0001 complete [ 892.241329] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 893.995572] Key type lgssc unregistered [ 894.167778] LNet: 38415:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 894.171670] LNetError: 38415:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 894.183918] LNet: Removed LNI 192.168.203.134@tcp [ 894.675639] Key type .llcrypt unregistered [ 894.677161] Key type ._llcrypt unregistered [ 909.592440] Key type ._llcrypt registered [ 909.594379] Key type .llcrypt registered [ 909.659245] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 910.441750] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 910.479874] alg: No test for adler32 (adler32-zlib) [ 911.448824] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 911.619561] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 913.264362] Key type lgssc registered [ 914.290481] Lustre: Echo OBD driver; http://www.lustre.org/ [ 918.835776] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 918.848662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 920.074444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 922.834910] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 928.528727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 928.757237] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 931.601720] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 936.014267] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 939.549507] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 943.913516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 944.191093] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 945.190910] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 948.950633] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 953.664550] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 960.481445] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 960.486457] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 960.502310] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 961.257637] Lustre: lustre-OST0000: Not available for connect from 192.168.203.34@tcp (stopping) [ 963.553662] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 963.555177] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 963.556050] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 965.303120] Lustre: server umount lustre-OST0000 complete [ 980.399592] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 980.414374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 980.422225] Lustre: Skipped 2 previous similar messages [ 986.623950] Lustre: server umount lustre-MDT0000 complete [ 988.642269] LustreError: 39807:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 988.654477] LustreError: 39807:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 988.710084] LustreError: 41649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786191832 with bad export cookie 12283357043760801089 [ 988.713904] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 988.719237] LustreError: 41649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 994.982065] Lustre: server umount lustre-MDT0001 complete [ 1000.887308] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 1002.480240] Key type lgssc unregistered [ 1002.706870] LNet: 43521:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1002.716176] LNetError: 43521:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1002.724859] LNet: Removed LNI 192.168.203.134@tcp [ 1003.252719] Key type .llcrypt unregistered [ 1003.257889] Key type ._llcrypt unregistered [ 1016.677138] Key type ._llcrypt registered [ 1016.678417] Key type .llcrypt registered [ 1016.732566] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 1017.191404] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1017.205475] alg: No test for adler32 (adler32-zlib) [ 1018.100168] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 1018.239281] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 1019.848264] Key type lgssc registered [ 1020.525609] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1024.991488] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1025.005396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1026.285860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1029.006557] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1034.263023] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1034.483583] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1037.354089] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1041.936660] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1045.157321] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1049.253108] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1049.453507] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1051.508901] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1053.555255] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1057.983170] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1064.930263] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1064.937488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1064.937992] Lustre: Skipped 2 previous similar messages [ 1064.945427] Lustre: Skipped 2 previous similar messages [ 1065.214949] Lustre: server umount lustre-MDT0000 complete [ 1065.712288] LustreError: 44909:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1065.726240] LustreError: 44909:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1067.373706] LustreError: 47490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786191911 with bad export cookie 12053645168491401590 [ 1067.375763] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1067.380249] LustreError: 47490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1070.048953] LustreError: 46181:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1070.049668] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1070.067683] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1072.097633] LustreError: 44907:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1072.098040] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1072.105735] LustreError: 44907:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1072.108943] Lustre: Skipped 1 previous similar message [ 1073.623954] Lustre: server umount lustre-MDT0001 complete [ 1092.217878] Lustre: server umount lustre-OST0000 complete [ 1101.361457] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 1103.360240] Key type lgssc unregistered [ 1103.557964] LNet: 48771:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1103.568297] LNetError: 48771:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1103.577817] LNet: Removed LNI 192.168.203.134@tcp [ 1104.049663] Key type .llcrypt unregistered [ 1104.051474] Key type ._llcrypt unregistered [ 1118.208288] Key type ._llcrypt registered [ 1118.210115] Key type .llcrypt registered [ 1118.292758] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 1118.847314] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1118.872568] alg: No test for adler32 (adler32-zlib) [ 1119.789930] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 1119.913524] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 1121.520164] Key type lgssc registered [ 1122.130672] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1126.223920] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1126.236438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1141.729410] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1146.848357] LustreError: 50135:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.134@tcp: failed processing log, type 4: rc = -110 [ 1173.472383] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1176.204384] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1179.770024] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1345.108906] Lustre: server umount lustre-OST0000 complete [ 1354.243314] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 1355.940317] Key type lgssc unregistered [ 1356.115629] LNet: 51812:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1356.124248] LNetError: 51812:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1356.138786] LNet: Removed LNI 192.168.203.134@tcp [ 1356.653977] Key type .llcrypt unregistered [ 1356.656917] Key type ._llcrypt unregistered [ 1369.923122] Key type ._llcrypt registered [ 1369.926088] Key type .llcrypt registered [ 1370.003734] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 1370.446248] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1370.455320] alg: No test for adler32 (adler32-zlib) [ 1371.348459] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 1371.480242] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 1373.096197] Key type lgssc registered [ 1373.680581] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1378.167569] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1378.186611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1379.408239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1381.657792] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1385.898240] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1386.092025] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1388.146937] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1392.071426] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1395.234442] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1398.849929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1399.039984] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1402.050689] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1404.145436] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1405.718484] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1409.506389] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1409.510816] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1409.515952] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1410.535129] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1410.541093] Lustre: Skipped 1 previous similar message [ 1412.974283] Lustre: server umount lustre-OST0000 complete [ 1415.651218] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1415.653226] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1415.658126] Lustre: Skipped 1 previous similar message [ 1415.671190] Lustre: Skipped 1 previous similar message [ 1420.770775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1420.774366] Lustre: Skipped 1 previous similar message [ 1421.437572] Lustre: server umount lustre-MDT0000 complete [ 1423.620495] LustreError: 53184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786192267 with bad export cookie 3581308470875882866 [ 1423.626158] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1423.635890] LustreError: 53184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1423.816508] Lustre: server umount lustre-MDT0001 complete [ 1429.691627] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 1431.410775] Key type lgssc unregistered [ 1431.612503] LNet: 56734:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1431.616632] LNetError: 56734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1431.628747] LNet: Removed LNI 192.168.203.134@tcp [ 1432.111683] Key type .llcrypt unregistered [ 1432.113331] Key type ._llcrypt unregistered [ 1446.152087] Key type ._llcrypt registered [ 1446.153672] Key type .llcrypt registered [ 1446.229979] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 1446.924069] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1446.944266] alg: No test for adler32 (adler32-zlib) [ 1447.869517] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 1448.012793] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 1449.632176] Key type lgssc registered [ 1450.364028] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1454.624210] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1454.638023] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1470.112431] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1475.232555] LustreError: 58098:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.134@tcp: failed processing log, type 4: rc = -110 [ 1501.856225] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1505.944800] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1510.473727] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1514.249510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1514.543101] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1514.592178] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1517.029727] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1521.436484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1521.638364] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1523.927316] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1527.877864] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1530.938788] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1534.945111] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1534.949280] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1534.956487] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1536.993858] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1536.995970] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1537.002182] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1538.138357] Lustre: server umount lustre-OST0000 complete [ 1555.474547] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1555.484381] Lustre: Skipped 1 previous similar message [ 1555.488773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1555.493671] Lustre: Skipped 2 previous similar messages [ 1557.537867] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1557.540716] Lustre: Skipped 1 previous similar message [ 1561.648513] Lustre: server umount lustre-MDT0000 complete [ 1562.594615] LustreError: 59134:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1562.616416] LustreError: 59134:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 1563.767069] LustreError: 58105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786192407 with bad export cookie 4568726499655350676 [ 1563.776452] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1569.999755] Lustre: server umount lustre-MDT0001 complete [ 1575.758201] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 1577.466455] Key type lgssc unregistered [ 1577.691948] LNet: 61845:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1577.698335] LNetError: 61845:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1577.706921] LNet: Removed LNI 192.168.203.134@tcp [ 1578.204135] Key type .llcrypt unregistered [ 1578.205689] Key type ._llcrypt unregistered [ 1591.603756] Key type ._llcrypt registered [ 1591.605980] Key type .llcrypt registered [ 1591.675735] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 1592.169362] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1592.191108] alg: No test for adler32 (adler32-zlib) [ 1593.148800] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 1593.309261] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 1594.936267] Key type lgssc registered [ 1595.515572] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1599.827583] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1599.841551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1601.075286] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1603.208711] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1607.545195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1607.760060] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1609.822864] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1613.709209] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1616.456898] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1619.920050] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1620.102878] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1623.204513] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1625.260890] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1627.213745] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1640.929119] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1640.932784] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1640.940972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1642.981271] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1642.981322] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1642.986088] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1646.757621] Lustre: server umount lustre-OST0000 complete [ 1648.609764] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1648.614683] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1648.622224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1648.625423] Lustre: Skipped 1 previous similar message [ 1653.218799] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1653.219867] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1653.229775] Lustre: Skipped 1 previous similar message [ 1654.726434] Lustre: server umount lustre-MDT0000 complete [ 1656.579630] LustreError: 65074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786192500 with bad export cookie 5962843263267499672 [ 1656.584194] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1656.591938] LustreError: 65074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1656.773369] Lustre: server umount lustre-MDT0001 complete [ 1662.573959] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 1664.033236] Key type lgssc unregistered [ 1664.199828] LNet: 66788:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1664.205927] LNetError: 66788:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1664.225238] LNet: Removed LNI 192.168.203.134@tcp [ 1664.798555] Key type .llcrypt unregistered [ 1664.803578] Key type ._llcrypt unregistered [ 1678.007339] Key type ._llcrypt registered [ 1678.008984] Key type .llcrypt registered [ 1678.076190] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 1678.556684] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1678.567501] alg: No test for adler32 (adler32-zlib) [ 1679.473955] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 1679.588658] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 1681.176216] Key type lgssc registered [ 1681.720781] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1685.783391] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 08:35:29 (1786192529) [ 1688.898825] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1688.908451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1690.099361] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1692.336207] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1696.780991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1696.991981] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1699.078960] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1702.454598] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1705.217584] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1708.460967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1708.627833] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1711.174606] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1714.519637] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1714.793710] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1722.871599] Lustre: Failing over lustre-MDT0000 [ 1723.088529] Lustre: server umount lustre-MDT0000 complete [ 1727.033553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1727.096374] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1727.160375] Lustre: *** cfs_fail_loc=135, val=0*** [ 1727.206771] LustreError: 71555:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 1727.210189] LustreError: 71555:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.134@tcp: cfg command failed: rc = -2 [ 1727.216721] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1727.216721] [ 1727.217989] LustreError: 68467:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1727.223633] LustreError: MGC192.168.203.134@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1727.232704] LustreError: 68467:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1727.239368] LustreError: 71543:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1727.247257] LustreError: 71543:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 1727.251794] LustreError: 71543:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 1727.254680] LustreError: 71543:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 1727.267750] Lustre: server umount lustre-MDT0000 complete [ 1727.269923] LustreError: 71543:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1727.459222] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1727.464395] Lustre: Skipped 1 previous similar message [ 1730.070349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1730.141985] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1730.280364] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1730.311166] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1732.076587] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1732.311676] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1735.659741] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1735.681303] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 1735.722076] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 1737.696848] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1737.701929] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1737.707065] Lustre: Skipped 1 previous similar message [ 1737.709715] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1740.770111] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1740.770712] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1740.776359] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1743.195896] Lustre: server umount lustre-OST0000 complete [ 1745.890202] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1745.890491] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1745.900951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1745.905054] Lustre: Skipped 2 previous similar messages [ 1750.878546] Lustre: server umount lustre-MDT0000 complete [ 1751.010666] LustreError: 68470:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1751.018903] LustreError: 68470:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 1752.519535] LustreError: 68452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786192596 with bad export cookie 2985641260149747242 [ 1752.523845] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1752.527256] LustreError: 68452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1752.656894] Lustre: server umount lustre-MDT0001 complete [ 1757.397371] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 1758.789584] Key type lgssc unregistered [ 1758.955876] LNet: 73334:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1758.966934] LNetError: 73334:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1758.983608] LNet: Removed LNI 192.168.203.134@tcp [ 1759.391579] Key type .llcrypt unregistered [ 1759.392993] Key type ._llcrypt unregistered [ 1770.184427] Key type ._llcrypt registered [ 1770.186033] Key type .llcrypt registered [ 1770.234844] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 1770.662729] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1770.703389] alg: No test for adler32 (adler32-zlib) [ 1771.612546] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 1771.724385] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 1773.320191] Key type lgssc registered [ 1773.736193] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1776.681119] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1776.688828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1777.864315] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1779.478836] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1783.108724] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1783.300430] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1785.454959] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1788.976142] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1791.574443] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1794.574523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1794.750708] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1797.431691] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1801.043123] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1802.986356] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 1814.896733] Lustre: Failing over lustre-MDT0000 [ 1815.067920] Lustre: server umount lustre-MDT0000 complete [ 1818.857909] LustreError: 74717:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1818.866890] LustreError: 74717:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1819.105577] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1819.117672] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1819.617820] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1819.620620] LustreError: 75449:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1819.624091] Lustre: Skipped 1 previous similar message [ 1819.635863] LustreError: 75449:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1819.738683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1819.814863] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1819.910076] Lustre: *** cfs_fail_loc=a05, val=0*** [ 1819.979925] LustreError: 77826:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 1819.985277] LustreError: 77826:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.134@tcp: cfg command failed: rc = -9 [ 1819.991343] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1819.991343] [ 1819.997388] LustreError: MGC192.168.203.134@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1820.009841] LustreError: 77814:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 1820.016993] LustreError: 77814:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 1820.022761] LustreError: 77814:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 1820.029069] LustreError: 77814:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 1820.048612] Lustre: server umount lustre-MDT0000 complete [ 1820.053581] LustreError: 77814:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 1823.534080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1823.603825] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1823.743496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1823.766875] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1823.961336] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1825.672794] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1828.842182] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1828.856280] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 1828.889471] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 1833.957271] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1833.958841] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1833.974764] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1833.982382] Lustre: Skipped 1 previous similar message [ 1836.515781] Lustre: server umount lustre-OST0000 complete [ 1839.074579] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1839.079046] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1839.081242] Lustre: Skipped 2 previous similar messages [ 1839.081521] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1839.091290] Lustre: Skipped 1 previous similar message [ 1844.193488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1844.197571] Lustre: Skipped 1 previous similar message [ 1844.788391] Lustre: server umount lustre-MDT0000 complete [ 1846.384918] LustreError: 74704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786192690 with bad export cookie 2253462693871175224 [ 1846.387924] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1846.392176] LustreError: 74704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1846.576676] Lustre: server umount lustre-MDT0001 complete [ 1851.888536] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 1853.464182] Key type lgssc unregistered [ 1853.620356] LNet: 79607:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1853.624582] LNetError: 79607:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1853.631671] LNet: Removed LNI 192.168.203.134@tcp [ 1854.087317] Key type .llcrypt unregistered [ 1854.088677] Key type ._llcrypt unregistered [ 1860.657950] Key type ._llcrypt registered [ 1860.660148] Key type .llcrypt registered [ 1860.727710] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 1860.902284] Key type .llcrypt unregistered [ 1860.904308] Key type ._llcrypt unregistered [ 1872.997812] Key type ._llcrypt registered [ 1872.999434] Key type .llcrypt registered [ 1873.056463] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 1873.557090] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1873.571535] alg: No test for adler32 (adler32-zlib) [ 1874.456638] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 1874.582793] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 1876.184166] Key type lgssc registered [ 1876.737305] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1880.577699] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1880.589460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1881.816331] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1883.869622] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1888.240601] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1888.443211] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1890.397317] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1893.977369] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1896.534713] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1899.967202] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1900.142831] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1902.949812] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1906.280929] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 1906.364399] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1922.017853] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1922.022060] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1922.030380] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1924.065177] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1924.065723] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1924.072357] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1926.182987] Lustre: server umount lustre-OST0000 complete [ 1929.186307] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1929.187643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1929.193492] Lustre: Skipped 1 previous similar message [ 1929.198866] Lustre: Skipped 2 previous similar messages [ 1934.119246] Lustre: server umount lustre-MDT0000 complete [ 1934.304934] LustreError: 82214:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1934.321312] LustreError: 82214:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1935.815930] LustreError: 84282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786192779 with bad export cookie 11838888719739933819 [ 1935.816859] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1935.822122] LustreError: 84282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1935.973417] Lustre: server umount lustre-MDT0001 complete [ 1940.604314] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 1941.729670] Key type lgssc unregistered [ 1941.863729] LNet: 85035:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1941.868542] LNetError: 85035:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1941.879590] LNet: Removed LNI 192.168.203.134@tcp [ 1942.230581] Key type .llcrypt unregistered [ 1942.232115] Key type ._llcrypt unregistered [ 1952.491710] Key type ._llcrypt registered [ 1952.492894] Key type .llcrypt registered [ 1952.541598] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 1953.004961] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1953.017909] alg: No test for adler32 (adler32-zlib) [ 1953.897963] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 1954.001637] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 1955.600294] Key type lgssc registered [ 1956.074843] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1959.574601] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1959.585476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1960.850251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1962.821841] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1966.921632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1967.102788] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1968.966768] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1972.476623] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1975.056823] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1978.206467] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1978.390764] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1981.254290] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 1984.872574] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1985.576381] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2000.867087] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2000.872772] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2000.896801] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2001.889774] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2001.892208] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2001.894415] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2007.009949] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2007.013346] Lustre: Skipped 2 previous similar messages [ 2012.128295] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2012.130340] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2012.140482] Lustre: Skipped 1 previous similar message [ 2012.219355] Lustre: server umount lustre-OST0000 complete [ 2017.250452] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2017.256017] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2017.259388] Lustre: Skipped 1 previous similar message [ 2017.265490] Lustre: Skipped 1 previous similar message [ 2020.410545] Lustre: server umount lustre-MDT0000 complete [ 2022.275555] LustreError: 86406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786192866 with bad export cookie 7689744970535291069 [ 2022.279316] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2022.289507] LustreError: 86406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2022.467667] Lustre: server umount lustre-MDT0001 complete [ 2027.820775] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 2029.335784] Key type lgssc unregistered [ 2029.507763] LNet: 89984:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2029.511956] LNetError: 89984:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2029.521472] LNet: Removed LNI 192.168.203.134@tcp [ 2030.034435] Key type .llcrypt unregistered [ 2030.039943] Key type ._llcrypt unregistered [ 2042.108231] Key type ._llcrypt registered [ 2042.110038] Key type .llcrypt registered [ 2042.156968] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 2042.595577] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2042.645684] alg: No test for adler32 (adler32-zlib) [ 2043.534421] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 2043.652282] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 2045.248146] Key type lgssc registered [ 2045.762567] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2049.277444] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2049.287148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2050.494570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2052.610518] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2056.971310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2057.171742] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2059.456928] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2063.400931] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2065.983879] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2069.280712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2069.440862] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2071.786068] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2074.754470] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2075.629832] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2090.979191] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2090.982627] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2090.995814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2093.027151] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2093.027398] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2093.036214] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2094.489914] Lustre: server umount lustre-OST0000 complete [ 2098.145595] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2098.147416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2098.151197] Lustre: Skipped 1 previous similar message [ 2098.151600] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2098.157110] Lustre: Skipped 2 previous similar messages [ 2102.297712] Lustre: server umount lustre-MDT0000 complete [ 2103.265164] LustreError: 92103:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2103.271781] LustreError: 92103:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2103.804728] LustreError: 91356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786192947 with bad export cookie 8047254507382869819 [ 2103.806751] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2103.810254] LustreError: 91356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2103.935544] Lustre: server umount lustre-MDT0001 complete [ 2108.142093] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 2109.303135] Key type lgssc unregistered [ 2109.435570] LNet: 94929:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2109.440689] LNetError: 94929:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2109.448584] LNet: Removed LNI 192.168.203.134@tcp [ 2109.766584] Key type .llcrypt unregistered [ 2109.767883] Key type ._llcrypt unregistered [ 2119.828421] Key type ._llcrypt registered [ 2119.830314] Key type .llcrypt registered [ 2119.879972] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 2120.336870] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2120.344814] alg: No test for adler32 (adler32-zlib) [ 2121.226022] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 2121.338000] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 2122.936393] Key type lgssc registered [ 2123.426123] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2126.839622] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2126.849330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2142.304327] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2147.424303] LustreError: 96292:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.134@tcp: failed processing log, type 4: rc = -110 [ 2174.048178] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2176.446232] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2179.580727] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2182.064505] Lustre: server umount lustre-OST0000 complete [ 2185.543408] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 08:43:49 (1786193029) [ 2188.567646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2188.774127] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2190.579706] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2193.889364] LustreError: 97928:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2193.896072] LustreError: 97928:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2194.624081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2194.817812] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2196.737201] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2199.952520] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2202.308224] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2205.156975] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2205.316426] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2208.449180] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2211.954901] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2213.549254] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 2229.217239] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2229.221243] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2229.229386] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2230.241642] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2230.242843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2230.245507] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2232.479877] Lustre: server umount lustre-OST0000 complete [ 2235.362391] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2235.363296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2235.368214] Lustre: Skipped 1 previous similar message [ 2235.371883] Lustre: Skipped 1 previous similar message [ 2240.414425] Lustre: server umount lustre-MDT0000 complete [ 2240.482158] LustreError: 97928:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2242.054939] LustreError: 97908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786193085 with bad export cookie 12885196755907097755 [ 2242.057959] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2242.062660] LustreError: 97908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2242.219442] Lustre: server umount lustre-MDT0001 complete [ 2247.292169] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 2248.859599] Key type lgssc unregistered [ 2249.038582] LNet: 101637:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2249.042362] LNetError: 101637:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2249.049546] LNet: Removed LNI 192.168.203.134@tcp [ 2249.486217] Key type .llcrypt unregistered [ 2249.488286] Key type ._llcrypt unregistered [ 2267.834602] Key type ._llcrypt registered [ 2267.836337] Key type .llcrypt registered [ 2267.889617] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 2268.265342] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2268.288287] alg: No test for adler32 (adler32-zlib) [ 2269.200944] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 2269.313817] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 2270.912172] Key type lgssc registered [ 2271.459328] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2274.893306] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2274.900562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2276.067787] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2278.004871] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2281.881259] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2282.048451] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2283.877085] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2287.032396] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2289.559133] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2292.030260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2292.164867] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2294.444113] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2297.478537] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2300.391332] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 2315.745114] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2315.747731] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2315.749051] LustreError: Skipped 1 previous similar message [ 2315.754886] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2316.770203] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2316.770940] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2316.771523] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2316.773034] Lustre: Skipped 1 previous similar message [ 2318.181072] Lustre: server umount lustre-OST0000 complete [ 2321.890024] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2321.890790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2321.894966] Lustre: Skipped 1 previous similar message [ 2321.899559] Lustre: Skipped 1 previous similar message [ 2325.978805] Lustre: server umount lustre-MDT0000 complete [ 2327.010445] LustreError: 103795:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2327.020528] LustreError: 103795:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2327.361422] LustreError: 103780:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786193171 with bad export cookie 11314642398779748006 [ 2327.366176] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2327.368190] LustreError: 103780:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2327.493747] Lustre: server umount lustre-MDT0001 complete [ 2331.565261] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 2332.706470] Key type lgssc unregistered [ 2332.842759] LNet: 107356:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2332.846741] LNetError: 107356:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2332.858430] LNet: Removed LNI 192.168.203.134@tcp [ 2333.200189] Key type .llcrypt unregistered [ 2333.202049] Key type ._llcrypt unregistered [ 2341.279382] Key type ._llcrypt registered [ 2341.280878] Key type .llcrypt registered [ 2341.325288] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 2341.754018] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2341.762343] alg: No test for adler32 (adler32-zlib) [ 2342.636116] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 2342.733873] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 2344.320264] Key type lgssc registered [ 2344.797025] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2347.924948] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2347.934585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2363.360785] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2368.480228] LustreError: 108576:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.134@tcp: failed processing log, type 4: rc = -110 [ 2395.104254] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2397.117513] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2399.712820] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2402.284443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2402.396465] LustreError: MGC192.168.203.134@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2402.402701] LustreError: 109597:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2402.406248] LustreError: 109597:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2402.409923] LustreError: 109597:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2402.448316] Lustre: server umount lustre-MDT0000 complete [ 2402.450206] LustreError: 109597:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2415.626420] Lustre: server umount lustre-OST0000 complete [ 2419.240234] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 2422.590835] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 2435.172132] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2435.254377] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2435.265906] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2435.302498] Lustre: lustre-MDT0000: new disk, initializing [ 2435.326611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2435.333976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2436.749434] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2441.132242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2441.179946] Lustre: 112384:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2441.207804] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2441.210366] Lustre: Skipped 1 previous similar message [ 2441.245181] Lustre: lustre-MDT0001: new disk, initializing [ 2441.283307] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2441.298123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2441.301821] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2442.714172] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2446.616559] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2448.590104] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2450.913791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2451.021970] Lustre: lustre-OST0000: new disk, initializing [ 2451.023802] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2451.027383] Lustre: 114350:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2451.060916] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2451.356892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2451.361378] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2451.399023] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2453.160068] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2457.117121] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2458.684160] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2458.770112] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2460.200725] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2460.280939] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2461.666869] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2461.668154] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2461.670310] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2461.673517] Lustre: Skipped 1 previous similar message [ 2462.994911] Lustre: server umount lustre-OST0000 complete [ 2466.785879] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2466.786472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2466.789883] Lustre: Skipped 1 previous similar message [ 2466.791842] Lustre: Skipped 1 previous similar message [ 2466.793373] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2466.797321] LustreError: Skipped 1 previous similar message [ 2470.297121] Lustre: server umount lustre-MDT0000 complete [ 2471.604302] LustreError: 112377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786193315 with bad export cookie 2509455689276562805 [ 2471.608282] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2471.609708] LustreError: 112377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2471.728872] Lustre: server umount lustre-MDT0001 complete [ 2474.447566] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 08:48:38 (1786193318) [ 2475.610090] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 2481.493000] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 2485.027617] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 2498.510071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2498.600519] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2498.612786] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2498.650881] Lustre: lustre-MDT0000: new disk, initializing [ 2498.680075] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2498.688310] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2500.172577] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2504.834214] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2504.874748] Lustre: 119240:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2504.899339] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2504.902358] Lustre: Skipped 1 previous similar message [ 2504.939598] Lustre: lustre-MDT0001: new disk, initializing [ 2504.989312] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2504.993956] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2506.387940] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2510.480797] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2512.579635] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2515.089830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2515.194522] Lustre: lustre-OST0000: new disk, initializing [ 2515.196920] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2515.200796] Lustre: 121205:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2515.236576] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2515.239341] Lustre: Skipped 1 previous similar message [ 2517.273249] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2521.179268] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2522.713888] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2524.669271] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2524.673845] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2524.708815] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2524.838658] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 2526.388054] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2526.478152] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2529.761895] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2529.762145] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2529.765417] LustreError: Skipped 1 previous similar message [ 2529.771674] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2529.775031] Lustre: Skipped 1 previous similar message [ 2533.338697] Lustre: server umount lustre-OST0000 complete [ 2534.881507] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2534.882373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2534.889571] Lustre: Skipped 2 previous similar messages [ 2540.001181] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2540.004121] Lustre: Skipped 2 previous similar messages [ 2549.216166] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2549.217334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2549.223008] Lustre: Skipped 3 previous similar messages [ 2549.272819] LustreError: 122756:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x5:0x0].0x0 (00000000b5ab9374) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 2549.306397] Lustre: server umount lustre-MDT0000 complete [ 2550.653570] LustreError: 119231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786193394 with bad export cookie 2509455689276563680 [ 2550.657597] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2550.658028] LustreError: 119231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2550.787542] Lustre: server umount lustre-MDT0001 complete [ 2553.882110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2554.061887] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2555.465306] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2558.453604] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2559.988528] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2562.521187] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2564.502227] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2566.911475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2569.033493] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2571.498973] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2584.617588] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 2588.644909] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2588.645727] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2588.646174] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2588.649124] Lustre: Skipped 1 previous similar message [ 2591.700249] Lustre: server umount lustre-OST0000 complete [ 2599.338052] Lustre: server umount lustre-MDT0000 complete [ 2600.814193] LustreError: 123415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786193444 with bad export cookie 2509455689276564548 [ 2600.817963] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2600.818967] LustreError: 123415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2600.942655] Lustre: server umount lustre-MDT0001 complete [ 2605.119785] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 2606.272835] Key type lgssc unregistered [ 2606.426662] LNet: 127079:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2606.430835] LNetError: 127079:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2606.440205] LNet: Removed LNI 192.168.203.134@tcp [ 2606.812666] Key type .llcrypt unregistered [ 2606.814245] Key type ._llcrypt unregistered [ 2613.990839] Key type ._llcrypt registered [ 2613.992523] Key type .llcrypt registered [ 2614.042997] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 2620.635601] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 2621.062363] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2621.070960] alg: No test for adler32 (adler32-zlib) [ 2621.943815] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 2622.051101] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 2623.640170] Key type lgssc registered [ 2624.035154] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2636.396958] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2636.402847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2637.489601] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2637.500904] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2637.534399] Lustre: lustre-MDT0000: new disk, initializing [ 2637.556393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2637.564597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2638.912815] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2643.221522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2643.268531] Lustre: 130208:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2643.294960] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2643.297370] Lustre: Skipped 1 previous similar message [ 2643.333921] Lustre: lustre-MDT0001: new disk, initializing [ 2643.373556] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2643.389297] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2643.393549] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2644.904934] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2648.723979] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2650.633115] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2653.101256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2653.221965] Lustre: lustre-OST0000: new disk, initializing [ 2653.224944] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2653.229652] Lustre: 132172:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2653.270443] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2655.563539] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2659.515414] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2660.877488] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2660.882185] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2660.918760] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2661.073990] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2661.157932] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2662.697112] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2662.777458] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2665.954328] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2665.955825] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2665.958102] LustreError: Skipped 1 previous similar message [ 2665.958879] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2665.963228] Lustre: Skipped 1 previous similar message [ 2669.588488] Lustre: server umount lustre-OST0000 complete [ 2671.073576] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2671.074513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2671.078812] Lustre: Skipped 1 previous similar message [ 2671.081100] Lustre: Skipped 1 previous similar message [ 2676.193238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2676.196110] Lustre: Skipped 2 previous similar messages [ 2677.149169] Lustre: server umount lustre-MDT0000 complete [ 2678.481230] LustreError: 130242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786193522 with bad export cookie 14062234671566963103 [ 2678.484282] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2678.486191] LustreError: 130242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2678.611516] Lustre: server umount lustre-MDT0001 complete [ 2681.460229] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 08:52:05 (1786193525) [ 2683.652455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2683.830499] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2685.262659] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2688.114096] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2688.237607] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2689.715414] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2692.279053] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2694.221887] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2698.721707] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2698.721819] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2698.722537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2698.727385] Lustre: Skipped 1 previous similar message [ 2701.078683] Lustre: server umount lustre-MDT0000 complete [ 2702.429707] LustreError: 134561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786193546 with bad export cookie 14062234671566963985 [ 2702.432478] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2702.435118] LustreError: 134561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2702.550143] Lustre: server umount lustre-MDT0001 complete [ 2705.413944] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 08:52:29 (1786193549) [ 2707.662918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2723.104758] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2728.224429] LustreError: 136991:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.134@tcp: failed processing log, type 4: rc = -110 [ 2753.824475] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2756.030647] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2758.946786] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2759.891031] Lustre: server umount lustre-OST0000 complete [ 2763.015491] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 08:53:26 (1786193606) [ 2765.823797] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2766.037281] LustreError: 138433:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2766.046547] LustreError: 138433:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2767.784520] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2771.091699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2771.227670] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2771.230532] Lustre: Skipped 1 previous similar message [ 2772.797828] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2775.723561] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2777.896857] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2780.563335] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2782.935727] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2785.804457] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2796.002360] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2796.003057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2796.008310] Lustre: Skipped 2 previous similar messages [ 2796.011313] Lustre: Skipped 4 previous similar messages [ 2798.833854] Lustre: server umount lustre-MDT0000 complete [ 2800.448192] LustreError: 139168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786193644 with bad export cookie 14062234671566964636 [ 2800.450977] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2800.453508] LustreError: 139168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2800.590676] Lustre: server umount lustre-MDT0001 complete [ 2812.380337] Lustre: server umount lustre-OST0000 complete [ 2815.513741] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 08:54:19 (1786193659) [ 2818.006649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2818.178201] LustreError: 142084:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2818.207571] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2818.211628] Lustre: Skipped 1 previous similar message [ 2819.649132] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2822.584525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2824.156263] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2826.683694] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2828.612363] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2830.955611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2833.014216] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2835.358720] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2836.729599] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2843.365444] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 2843.943753] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 7 sec [ 2845.410290] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2845.500182] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2848.738430] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2848.739372] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2848.741105] LustreError: Skipped 1 previous similar message [ 2848.747420] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2848.750621] Lustre: Skipped 2 previous similar messages [ 2852.307701] Lustre: server umount lustre-OST0000 complete [ 2858.465287] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2858.468742] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2858.474795] Lustre: Skipped 2 previous similar messages [ 2859.678617] Lustre: server umount lustre-MDT0000 complete [ 2861.000607] LustreError: 142819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786193704 with bad export cookie 14062234671566965707 [ 2861.005633] LustreError: 142819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2861.005872] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2861.121826] Lustre: server umount lustre-MDT0001 complete [ 2863.983374] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 08:55:07 (1786193707) [ 2866.319673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2881.760598] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2886.881292] LustreError: 146289:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.134@tcp: failed processing log, type 4: rc = -110 [ 2912.480290] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2912.483088] Lustre: Skipped 2 previous similar messages [ 2914.575336] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2917.244586] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2919.583779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2919.734362] LustreError: 146314:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2919.789036] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 2921.311865] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2924.328374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2925.934528] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 2928.674162] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2930.692022] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2932.231830] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2932.319546] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2933.884118] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2933.966140] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2934.753173] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2934.756237] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2934.761996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2934.764954] Lustre: Skipped 3 previous similar messages [ 2940.757342] Lustre: server umount lustre-MDT0000 complete [ 2941.952513] LustreError: 146296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786193785 with bad export cookie 14062234671566966715 [ 2941.956286] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2941.957737] LustreError: 146296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2942.070672] Lustre: server umount lustre-MDT0001 complete [ 2953.625772] Lustre: server umount lustre-OST0000 complete [ 2956.390728] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 08:56:40 (1786193800) [ 2958.505586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2973.920526] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2979.040370] LustreError: 150480:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.134@tcp: failed processing log, type 4: rc = -110 [ 3006.641806] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3009.054885] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3011.308045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3011.452944] LustreError: 150505:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3011.503094] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 3012.892755] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3015.941081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3017.530536] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3019.968961] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3021.876767] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3024.348242] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3024.420270] Lustre: lustre-OST0001: new disk, initializing [ 3024.422813] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3024.426213] Lustre: 153350:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3026.659445] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3027.267139] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3027.272884] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3027.310034] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3030.848316] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3032.433770] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3032.523871] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3034.156257] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3034.248769] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3036.641319] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3036.645117] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3036.649896] Lustre: Skipped 2 previous similar messages [ 3036.652163] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3036.654269] Lustre: Skipped 3 previous similar messages [ 3041.084881] Lustre: server umount lustre-OST0000 complete [ 3042.788672] LustreError: 151555:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3042.794644] LustreError: 151555:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3048.469888] Lustre: server umount lustre-OST0001 complete [ 3064.288118] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3064.352664] Lustre: server umount lustre-MDT0000 complete [ 3065.673532] LustreError: 150487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786193909 with bad export cookie 14062234671566967667 [ 3065.676348] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3065.678866] LustreError: 150487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3065.800319] Lustre: server umount lustre-MDT0001 complete [ 3073.682401] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 08:58:37 (1786193917) [ 3074.189603] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 3074.720531] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 3075.227636] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 08:58:39 (1786193919) [ 3077.297732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3077.379189] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3077.391732] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3077.460366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3077.463103] Lustre: Skipped 6 previous similar messages [ 3078.681290] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3081.357439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3081.389080] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3081.401543] Lustre: 156901:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3082.811442] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3084.997804] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3086.617182] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3088.630216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3088.689953] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3090.453819] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3091.017794] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 3092.710217] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3094.005201] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3094.067739] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3095.274919] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3095.342076] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3096.034222] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3096.036889] LustreError: Skipped 5 previous similar messages [ 3098.002465] Lustre: server umount lustre-OST0000 complete [ 3111.810304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3114.138253] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3116.741052] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3117.090544] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3117.096201] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3117.096448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3118.326812] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3118.401668] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3119.824549] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3119.901562] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3121.626474] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9eca89e68000.ost_server_uuid 50 [ 3122.141246] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9eca89e68000.ost_server_uuid in FULL state after 0 sec [ 3122.201209] LustreError: 156910:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.203.34@tcp) returned error from blocking AST (req@00000000b7d94a22 x1872959598942720 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 000000002a90dd90/0xc32717edcaee5cbf lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.34@tcp remote: 0x777b6d7ddc03f728 expref: 5 pid: 156910 timeout: 3222 lvb_type: 0 lru_score: 0 lru_type: 0 [ 3122.213537] LustreError: lustre-MDT0000: A client on nid 192.168.203.34@tcp was evicted due to a lock blocking callback time out: rc -107 [ 3122.216906] LustreError: 156899:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.203.34@tcp ns: mdt-lustre-MDT0000_UUID lock: 000000002a90dd90/0xc32717edcaee5cbf lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.34@tcp remote: 0x777b6d7ddc03f728 expref: 6 pid: 156910 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 3127.266910] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3127.271343] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3127.272076] Lustre: Skipped 8 previous similar messages [ 3127.273820] Lustre: Skipped 15 previous similar messages [ 3129.361852] Lustre: server umount lustre-OST0000 complete [ 3136.795683] Lustre: server umount lustre-MDT0000 complete [ 3137.505185] LustreError: 156915:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3137.512492] LustreError: 156915:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3137.939234] LustreError: 156939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786193981 with bad export cookie 14062234671566968850 [ 3137.942020] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3137.942708] LustreError: 156939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3152.352087] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3152.436867] Lustre: server umount lustre-MDT0001 complete [ 3155.675704] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 3156.625536] Key type lgssc unregistered [ 3156.745334] LNet: 163008:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3156.747757] LNetError: 163008:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3156.754260] LNet: Removed LNI 192.168.203.134@tcp [ 3157.030360] Key type .llcrypt unregistered [ 3157.031480] Key type ._llcrypt unregistered [ 3165.316208] Key type ._llcrypt registered [ 3165.317142] Key type .llcrypt registered [ 3165.351445] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 3165.652963] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3165.681561] alg: No test for adler32 (adler32-zlib) [ 3166.540367] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 3166.619709] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 3168.192148] Key type lgssc registered [ 3168.512196] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3170.854754] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3170.859812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3171.983666] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3173.134622] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3175.641640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3175.748414] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3176.899123] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3179.049266] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3180.746342] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3182.749535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3182.844693] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3184.661176] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3186.838838] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3196.198578] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 3199.194432] Lustre: Failing over lustre-MDT0000 [ 3199.351915] Lustre: server umount lustre-MDT0000 complete [ 3201.507022] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3201.507767] LustreError: 164398:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3201.512209] Lustre: Skipped 1 previous similar message [ 3201.519627] LustreError: 164398:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3201.522736] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3206.625876] LustreError: 165136:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3206.630952] LustreError: 165136:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3210.205618] LustreError: 164380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.34@tcp arrived at 1786194054 with bad export cookie 10131594150340578745 [ 3210.211944] LustreError: 164380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3211.746932] LustreError: 164394:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3211.755906] LustreError: 164394:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3212.679426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3212.721968] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3212.806249] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3212.835799] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3214.198670] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3214.930257] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3214.933563] Lustre: lustre-MDT0000: Denying connection for new client 81f69da4-004f-4c5d-81a1-446727fbe2ce (at 192.168.203.34@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 3217.894538] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3221.984923] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3221.988407] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3221.995123] Lustre: Skipped 1 previous similar message [ 3221.998357] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3223.010799] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3223.011358] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3227.792117] Lustre: server umount lustre-OST0000 complete [ 3228.869137] LustreError: 168156:0:(ldlm_lib.c:3004:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3228.873317] Lustre: 167492:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3228.877507] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 3228.883173] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 3228.887558] LustreError: 167492:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000a80ac1a8 x1872960170265344/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3228.893291] LustreError: 167492:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 3228.896397] LustreError: 167492:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 3228.899544] Lustre: lustre-MDT0000: Recovery over after 0:14, of 2 clients 0 recovered and 2 were evicted. [ 3228.993769] Lustre: server umount lustre-MDT0000 complete [ 3230.089114] LustreError: 166992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786194073 with bad export cookie 10131594150340579102 [ 3230.092302] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3230.093828] LustreError: 166992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3230.097885] LustreError: 163707:0:(import.c:707:ptlrpc_connect_import_locked()) can't connect to a closed import [ 3236.270586] Lustre: server umount lustre-MDT0001 complete [ 3239.773420] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 3240.656686] Key type lgssc unregistered [ 3240.765325] LNet: 168909:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3240.767408] LNetError: 168909:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3240.777280] LNet: Removed LNI 192.168.203.134@tcp [ 3241.030424] Key type .llcrypt unregistered [ 3241.031363] Key type ._llcrypt unregistered [ 3249.468546] Key type ._llcrypt registered [ 3249.469614] Key type .llcrypt registered [ 3249.508384] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 3249.830060] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3249.890590] alg: No test for adler32 (adler32-zlib) [ 3250.749512] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 3250.848990] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 3252.432146] Key type lgssc registered [ 3252.823380] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3255.563617] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3255.570187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3256.736347] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3258.061755] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3260.860592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3260.974484] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3262.143690] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3264.247633] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3265.822984] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3267.830689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3267.952545] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3269.898284] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3272.193438] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3281.255865] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 3286.499955] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3286.500293] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3286.502497] LustreError: Skipped 1 previous similar message [ 3286.507117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3286.510534] Lustre: Skipped 1 previous similar message [ 3289.998069] Lustre: server umount lustre-OST0000 complete [ 3291.618778] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3291.618931] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3291.622963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3291.629177] Lustre: Skipped 2 previous similar messages [ 3291.634283] Lustre: Skipped 1 previous similar message [ 3296.737353] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3296.739979] Lustre: Skipped 1 previous similar message [ 3297.376852] Lustre: server umount lustre-MDT0000 complete [ 3298.611264] LustreError: 171036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786194142 with bad export cookie 1317646601291969486 [ 3298.614615] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3298.615484] LustreError: 171036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3298.744555] Lustre: server umount lustre-MDT0001 complete [ 3302.327461] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 3303.372798] Key type lgssc unregistered [ 3303.498296] LNet: 173856:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3303.500984] LNetError: 173856:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3303.512325] LNet: Removed LNI 192.168.203.134@tcp [ 3303.848822] Key type .llcrypt unregistered [ 3303.850045] Key type ._llcrypt unregistered [ 3315.325827] Key type ._llcrypt registered [ 3315.327665] Key type .llcrypt registered [ 3315.363295] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 3315.723340] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3315.751526] alg: No test for adler32 (adler32-zlib) [ 3316.608733] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 3316.701969] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 3318.280168] Key type lgssc registered [ 3318.667370] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3321.318841] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3321.326665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3322.471863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3323.798568] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3326.700493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3326.815275] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3328.083442] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3330.353238] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3332.122421] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3334.309848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3334.417672] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3336.167399] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3338.342595] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3347.750234] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 3352.032840] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3352.036382] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3352.042035] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3353.058390] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3353.059790] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3353.060505] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3353.069415] Lustre: Skipped 1 previous similar message [ 3355.927261] Lustre: server umount lustre-OST0000 complete [ 3357.664895] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3357.668949] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3357.674286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3363.297635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3363.299557] Lustre: Skipped 3 previous similar messages [ 3363.365276] Lustre: server umount lustre-MDT0000 complete [ 3364.674770] LustreError: 178032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786194208 with bad export cookie 8339437908725749120 [ 3364.679980] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3364.682842] LustreError: 178032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3364.816817] Lustre: server umount lustre-MDT0001 complete [ 3368.390530] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 3369.441762] Key type lgssc unregistered [ 3369.554392] LNet: 179088:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3369.556951] LNetError: 179088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3369.564291] LNet: Removed LNI 192.168.203.134@tcp [ 3369.866913] Key type .llcrypt unregistered [ 3369.868952] Key type ._llcrypt unregistered [ 3378.409354] Key type ._llcrypt registered [ 3378.410623] Key type .llcrypt registered [ 3378.447710] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 3378.790885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3378.841963] alg: No test for adler32 (adler32-zlib) [ 3379.700274] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 3379.792445] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 3381.368134] Key type lgssc registered [ 3381.699886] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3385.217567] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3385.224683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3386.354652] Lustre: *** cfs_fail_loc=135, val=0*** [ 3386.397731] LustreError: 180587:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 3386.400300] LustreError: 180587:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.134@tcp: cfg command failed: rc = -2 [ 3386.402933] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 3386.402933] [ 3386.405673] LustreError: MGC192.168.203.134@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3386.409930] LustreError: 180548:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3386.412323] LustreError: 180548:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3386.414737] LustreError: 180548:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3386.417070] LustreError: 180548:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 3386.452649] Lustre: server umount lustre-MDT0000 complete [ 3386.453989] LustreError: 180548:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3388.714293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3388.858102] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3390.193408] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3393.008106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3393.127213] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3394.441534] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3396.823348] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3398.650068] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3402.401803] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3402.402834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3402.405693] Lustre: Skipped 1 previous similar message [ 3405.463921] Lustre: server umount lustre-MDT0000 complete [ 3406.676209] LustreError: 181692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786194250 with bad export cookie 10613515522343799139 [ 3406.679503] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3406.682164] LustreError: 181692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3406.788194] Lustre: server umount lustre-MDT0001 complete [ 3410.438220] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 3411.576645] Key type lgssc unregistered [ 3411.726446] LNet: 183277:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3411.729996] LNetError: 183277:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3411.736327] LNet: Removed LNI 192.168.203.134@tcp [ 3412.061965] Key type .llcrypt unregistered [ 3412.064048] Key type ._llcrypt unregistered [ 3418.976562] Key type ._llcrypt registered [ 3418.977712] Key type .llcrypt registered [ 3419.014593] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 3419.128204] Key type .llcrypt unregistered [ 3419.129437] Key type ._llcrypt unregistered [ 3425.882714] Key type ._llcrypt registered [ 3425.883833] Key type .llcrypt registered [ 3425.928353] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 3426.297603] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3426.341091] alg: No test for adler32 (adler32-zlib) [ 3427.225075] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 3427.315434] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 3428.896153] Key type lgssc registered [ 3429.260727] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3431.786544] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3431.792616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3447.200419] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3452.320246] LustreError: 185388:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.134@tcp: failed processing log, type 4: rc = -110 [ 3478.944177] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3480.836283] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3483.371468] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3485.637312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3485.808257] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3485.838785] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 3487.199020] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3490.080205] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3490.215694] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3491.602321] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3494.147554] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3496.036498] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3496.944705] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 3503.587291] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3503.594077] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3503.597023] Lustre: Skipped 1 previous similar message [ 3508.707363] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3508.709722] Lustre: Skipped 1 previous similar message [ 3509.072462] Lustre: server umount lustre-OST0000 complete [ 3510.753221] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3510.757428] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3510.764167] Lustre: Skipped 1 previous similar message [ 3510.767195] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3513.825387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3513.826038] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3516.381556] Lustre: server umount lustre-MDT0000 complete [ 3517.684180] LustreError: 185394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786194361 with bad export cookie 4224095902021916663 [ 3517.688386] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3517.689691] LustreError: 185394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3517.807990] Lustre: server umount lustre-MDT0001 complete [ 3521.491679] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 3522.549232] Key type lgssc unregistered [ 3522.684401] LNet: 189351:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3522.687049] LNetError: 189351:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3522.694270] LNet: Removed LNI 192.168.203.134@tcp [ 3523.001337] Key type .llcrypt unregistered [ 3523.003114] Key type ._llcrypt unregistered [ 3532.062904] Key type ._llcrypt registered [ 3532.064386] Key type .llcrypt registered [ 3532.104363] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 3532.457694] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3532.496488] alg: No test for adler32 (adler32-zlib) [ 3533.348030] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 3533.452052] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 3535.040135] Key type lgssc registered [ 3535.409158] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3538.018161] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3538.024289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3539.168492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3540.580319] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3543.377899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3543.519642] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3544.902972] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3547.417817] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3549.271093] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3551.550263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3551.692041] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3553.609320] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3555.992271] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3565.031753] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 3567.470612] Lustre: Failing over lustre-MDT0000 [ 3567.565847] Lustre: server umount lustre-MDT0000 complete [ 3569.121279] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3569.123831] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3569.128127] LustreError: 190739:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3569.132665] LustreError: 190739:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3570.145386] LustreError: 190740:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3570.146592] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3571.424854] LustreError: 190735:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3571.432138] LustreError: 190735:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3575.265094] LustreError: 191468:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3575.269924] LustreError: 191468:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3580.385296] LustreError: 190739:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3580.391738] LustreError: 190739:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3580.551693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3580.593426] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3580.680840] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3580.701216] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3581.656332] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3582.081816] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3583.653450] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 3586.023379] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3586.031378] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 3586.048374] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 3592.144957] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 3599.840907] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3599.845196] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3599.851707] Lustre: Skipped 1 previous similar message [ 3599.854808] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3601.379234] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3601.380128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3601.385605] Lustre: Skipped 1 previous similar message [ 3604.560338] Lustre: server umount lustre-OST0000 complete [ 3606.498037] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3606.498169] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3606.502545] Lustre: Skipped 1 previous similar message [ 3606.503485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3606.508648] Lustre: Skipped 1 previous similar message [ 3611.617109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3611.619345] Lustre: Skipped 1 previous similar message [ 3616.738969] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3616.742120] Lustre: Skipped 1 previous similar message [ 3620.320144] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3620.377078] Lustre: server umount lustre-MDT0000 complete [ 3621.636169] LustreError: 190721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786194465 with bad export cookie 14733062844043273545 [ 3621.639160] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3621.641171] LustreError: 190721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3621.769238] Lustre: server umount lustre-MDT0001 complete [ 3625.275306] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 3626.353990] Key type lgssc unregistered [ 3626.485415] LNet: 195789:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3626.488749] LNetError: 195789:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3626.499346] LNet: Removed LNI 192.168.203.134@tcp [ 3626.845238] Key type .llcrypt unregistered [ 3626.846548] Key type ._llcrypt unregistered [ 3635.300012] Key type ._llcrypt registered [ 3635.300984] Key type .llcrypt registered [ 3635.334693] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 3635.626723] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3635.635695] alg: No test for adler32 (adler32-zlib) [ 3636.480547] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 3636.559896] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 3638.144178] Key type lgssc registered [ 3638.473828] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3640.798435] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3640.804689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3641.939415] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3643.086408] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3645.669968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3645.780244] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3647.145226] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3649.598424] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3651.475646] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3653.645824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3653.759968] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3655.646895] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3658.014415] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3667.114390] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 3669.324189] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 3675.852346] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 3684.355670] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3693.027590] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3693.028055] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3693.028788] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3693.032022] Lustre: Skipped 1 previous similar message [ 3698.148263] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3698.151374] Lustre: Skipped 2 previous similar messages [ 3698.645760] Lustre: server umount lustre-OST0000 complete [ 3702.240915] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3702.243606] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3702.247711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3703.267369] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3706.080420] Lustre: server umount lustre-MDT0000 complete [ 3707.363073] LustreError: 197158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786194551 with bad export cookie 10281283284451644012 [ 3707.366733] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3707.367756] LustreError: 197158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3707.505077] Lustre: server umount lustre-MDT0001 complete [ 3711.252251] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 3712.285418] Key type lgssc unregistered [ 3712.420390] LNet: 201042:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3712.423830] LNetError: 201042:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3712.433362] LNet: Removed LNI 192.168.203.134@tcp [ 3712.763316] Key type .llcrypt unregistered [ 3712.764612] Key type ._llcrypt unregistered [ 3720.045239] Key type ._llcrypt registered [ 3720.046329] Key type .llcrypt registered [ 3720.079839] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 3720.503105] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3720.510306] alg: No test for adler32 (adler32-zlib) [ 3721.396277] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 3721.498049] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 3723.080147] Key type lgssc registered [ 3723.446164] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3726.074968] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3726.082467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3727.226340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3728.574748] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3731.374030] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3731.490957] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3732.800120] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3735.181861] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3736.988317] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3739.278930] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3739.407933] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3741.385420] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3743.887427] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3752.746752] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 3755.643143] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 3764.530684] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 3771.279120] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 3781.602311] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 3788.770850] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3788.770992] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3788.776181] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3788.777693] Lustre: Skipped 1 previous similar message [ 3793.891045] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3793.894301] Lustre: Skipped 1 previous similar message [ 3794.066225] Lustre: server umount lustre-OST0000 complete [ 3797.984870] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3797.987368] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3797.991200] Lustre: Skipped 1 previous similar message [ 3797.992987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3799.010254] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3801.431508] Lustre: server umount lustre-MDT0000 complete [ 3802.680699] LustreError: 205918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786194646 with bad export cookie 15859997827407911156 [ 3802.684347] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3802.686740] LustreError: 205918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3802.817449] Lustre: server umount lustre-MDT0001 complete [ 3806.524086] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 3807.582530] Key type lgssc unregistered [ 3807.711437] LNet: 207753:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3807.714187] LNetError: 207753:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3807.726332] LNet: Removed LNI 192.168.203.134@tcp [ 3808.040471] Key type .llcrypt unregistered [ 3808.041667] Key type ._llcrypt unregistered [ 3816.871150] Key type ._llcrypt registered [ 3816.872171] Key type .llcrypt registered [ 3816.909327] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 3817.331136] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3817.341829] alg: No test for adler32 (adler32-zlib) [ 3818.198323] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 3818.289601] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 3819.864153] Key type lgssc registered [ 3820.195263] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3822.552822] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3822.559959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3823.681444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3824.927432] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3827.541681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3827.649886] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3828.823184] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3830.990234] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3832.617880] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3834.586747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3834.692801] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3836.499370] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3838.686533] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3849.063209] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 3853.282320] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3853.285067] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3853.291246] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3854.307286] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3854.307738] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3854.314096] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3854.315361] Lustre: Skipped 1 previous similar message [ 3858.706444] Lustre: server umount lustre-OST0000 complete [ 3863.520889] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3863.525108] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3863.529885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3869.665155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3869.667437] Lustre: Skipped 3 previous similar messages [ 3874.272145] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3874.340641] Lustre: server umount lustre-MDT0000 complete [ 3874.785567] LustreError: 211634:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3874.791604] LustreError: 211634:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3875.618097] LustreError: 209169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786194719 with bad export cookie 5415483895996242667 [ 3875.620966] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3875.622239] LustreError: 209169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3875.748220] Lustre: server umount lustre-MDT0001 complete [ 3879.386275] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 3880.410941] Key type lgssc unregistered [ 3880.543440] LNet: 212846:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3880.546225] LNetError: 212846:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3880.555377] LNet: Removed LNI 192.168.203.134@tcp [ 3880.865293] Key type .llcrypt unregistered [ 3880.867043] Key type ._llcrypt unregistered [ 3888.264324] Key type ._llcrypt registered [ 3888.266355] Key type .llcrypt registered [ 3888.305749] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 3888.702202] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3888.724498] alg: No test for adler32 (adler32-zlib) [ 3889.579807] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 3889.665928] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 3891.248167] Key type lgssc registered [ 3891.595815] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3894.087672] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3894.094934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3895.236412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3896.565037] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3899.568510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3899.700645] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3901.055437] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3903.504486] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3905.342101] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3907.595155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3907.717575] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3909.753479] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3912.280491] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3921.067706] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 3926.497650] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3926.500208] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3926.501037] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3926.503878] Lustre: Skipped 1 previous similar message [ 3926.508494] Lustre: Skipped 1 previous similar message [ 3927.964101] Lustre: server umount lustre-OST0000 complete [ 3930.592788] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3930.595122] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3930.598827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3931.355380] Lustre: server umount lustre-MDT0000 complete [ 3931.617193] LustreError: 214088:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3931.617900] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3931.626279] LustreError: 214088:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3932.600607] LustreError: 214068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786194776 with bad export cookie 5566296850531736141 [ 3932.604228] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3932.604383] LustreError: 214068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3932.730682] Lustre: server umount lustre-MDT0001 complete [ 3936.372164] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 3937.475380] Key type lgssc unregistered [ 3937.610489] LNet: 217788:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3937.614333] LNetError: 217788:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3937.624349] LNet: Removed LNI 192.168.203.134@tcp [ 3937.968682] Key type .llcrypt unregistered [ 3937.969853] Key type ._llcrypt unregistered [ 3945.434613] Key type ._llcrypt registered [ 3945.435675] Key type .llcrypt registered [ 3945.477233] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 3945.838075] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3945.880943] alg: No test for adler32 (adler32-zlib) [ 3946.727187] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 3946.807576] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 3948.392131] Key type lgssc registered [ 3948.697137] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3951.017666] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3951.023310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3952.144717] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3953.319501] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3956.102492] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3956.233843] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3957.481895] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3959.718204] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3961.466875] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3963.496519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3963.627272] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3965.457821] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3967.606826] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3977.956520] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 3985.310533] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3985.334061] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3985.399563] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3987.195409] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 3989.503617] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3991.970077] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec [ 4002.490437] Lustre: Permanently deactivating lustre-OST0001 [ 4002.496130] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 4009.974876] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 4013.149281] Lustre: Permanently reactivating lustre-OST0001 [ 4013.154064] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 4013.156060] Lustre: Skipped 2 previous similar messages [ 4016.244344] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4016.250493] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 4016.253420] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4016.256678] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4016.265920] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4025.316073] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4025.317413] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4025.319642] Lustre: Skipped 2 previous similar messages [ 4029.503863] Lustre: server umount lustre-OST0001 complete [ 4030.437668] LustreError: 222553:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4030.444609] LustreError: 222553:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4033.505325] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4033.509268] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4033.516282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4033.518731] Lustre: Skipped 1 previous similar message [ 4034.528986] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4034.531781] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4035.559150] LustreError: 220851:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4035.567626] LustreError: 220851:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4040.674665] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4040.676479] LustreError: 220851:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4040.677125] Lustre: Skipped 3 previous similar messages [ 4040.683659] LustreError: 220851:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4045.280117] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4045.347658] Lustre: server umount lustre-OST0000 complete [ 4048.352939] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4048.356560] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4048.361029] Lustre: Skipped 1 previous similar message [ 4048.363034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4052.708610] Lustre: server umount lustre-MDT0000 complete [ 4054.021584] LustreError: 219766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786194897 with bad export cookie 9508280239238475301 [ 4054.023674] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4054.025378] LustreError: 219766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4054.159047] Lustre: server umount lustre-MDT0001 complete [ 4058.028977] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 4059.136373] Key type lgssc unregistered [ 4059.272385] LNet: 224094:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4059.275270] LNetError: 224094:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4059.283341] LNet: Removed LNI 192.168.203.134@tcp [ 4059.614798] Key type .llcrypt unregistered [ 4059.615997] Key type ._llcrypt unregistered [ 4073.931504] Key type ._llcrypt registered [ 4073.932556] Key type .llcrypt registered [ 4073.974300] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 4074.316325] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4074.324389] alg: No test for adler32 (adler32-zlib) [ 4075.196046] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 4075.308155] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 4076.896172] Key type lgssc registered [ 4077.306698] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4080.056787] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4080.064608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4081.170674] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4081.179362] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4081.250567] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4082.662447] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4085.442755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4085.478655] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4085.490124] Lustre: 226275:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4085.593670] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4086.937401] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4089.355113] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4091.122527] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4093.315594] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4093.383916] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4093.446946] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4095.354897] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4096.272819] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 4097.772610] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4099.338583] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 4106.875647] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 4113.403235] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 4119.934315] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 4128.504513] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 4150.144582] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 4150.147324] Lustre: Skipped 2 previous similar messages [ 4183.508183] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 4183.510684] Lustre: Skipped 3 previous similar messages [ 4256.764582] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 4256.767121] Lustre: Skipped 8 previous similar messages [ 4268.514914] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4268.515962] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4268.520298] Lustre: Skipped 1 previous similar message [ 4268.523200] Lustre: Skipped 1 previous similar message [ 4273.635445] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4273.638988] Lustre: Skipped 1 previous similar message [ 4274.002431] Lustre: server umount lustre-OST0000 complete [ 4278.753848] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4278.754531] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4278.759470] Lustre: Skipped 1 previous similar message [ 4281.437447] Lustre: server umount lustre-MDT0000 complete [ 4282.719281] LustreError: 229285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786195126 with bad export cookie 13481450190328489335 [ 4282.721914] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4282.724826] LustreError: 229285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4282.850878] Lustre: server umount lustre-MDT0001 complete [ 4286.269624] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 4287.180218] Key type lgssc unregistered [ 4287.290300] LNet: 231107:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4287.292429] LNetError: 231107:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4287.299243] LNet: Removed LNI 192.168.203.134@tcp [ 4287.579235] Key type .llcrypt unregistered [ 4287.580249] Key type ._llcrypt unregistered [ 4296.148733] Key type ._llcrypt registered [ 4296.149751] Key type .llcrypt registered [ 4296.182406] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 4296.544231] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4296.557220] alg: No test for adler32 (adler32-zlib) [ 4297.405610] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 4297.481779] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 4299.056121] Key type lgssc registered [ 4299.388660] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4301.788205] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4301.794620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4302.928561] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4304.239804] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4306.769992] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4306.884139] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4308.104275] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4310.358531] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4312.060840] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4314.189973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4314.341881] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4316.098624] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_default_debug -1 all [ 4318.343911] Lustre: DEBUG MARKER: oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4327.654630] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 4337.120736] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4337.123199] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4337.127620] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4338.149959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4338.150001] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4338.154670] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4339.599083] Lustre: server umount lustre-OST0000 complete [ 4342.752869] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4342.755327] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4342.759754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4342.762443] Lustre: Skipped 1 previous similar message [ 4347.030482] Lustre: server umount lustre-MDT0000 complete [ 4348.300398] LustreError: 233229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786195192 with bad export cookie 13068214803012122691 [ 4348.303610] LustreError: MGC192.168.203.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4348.305847] LustreError: 233229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4348.427631] Lustre: server umount lustre-MDT0001 complete [ 4351.848584] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 4352.781746] Key type lgssc unregistered [ 4352.895293] LNet: 236202:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4352.897688] LNetError: 236202:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4352.905345] LNet: Removed LNI 192.168.203.134@tcp [ 4353.178533] Key type .llcrypt unregistered [ 4353.179526] Key type ._llcrypt unregistered [ 4361.321431] Key type ._llcrypt registered [ 4361.322665] Key type .llcrypt registered [ 4361.355920] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 4361.689677] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4361.710560] alg: No test for adler32 (adler32-zlib) [ 4362.548624] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 4362.626708] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 4364.200156] Key type lgssc registered [ 4364.520623] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4627.800580] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 4628.809904] Key type lgssc unregistered [ 4628.933383] LNet: 238205:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4628.935803] LNetError: 238205:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4628.943298] LNet: Removed LNI 192.168.203.134@tcp [ 4629.224832] Key type .llcrypt unregistered [ 4629.226073] Key type ._llcrypt unregistered [ 4637.144451] Key type ._llcrypt registered [ 4637.145807] Key type .llcrypt registered [ 4637.185955] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing set_hostid [ 4643.616843] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing load_modules_local [ 4643.933456] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4643.940171] alg: No test for adler32 (adler32-zlib) [ 4644.784038] Lustre: Lustre: Build Version: 2.17.56_51_geba2e8d [ 4644.871375] LNet: Added LNI 192.168.203.134@tcp [8/256/0/180] [ 4646.456121] Key type lgssc registered [ 4646.805473] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4657.048139] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 4349 sec ========== 09:25:00 (1786195500) [ 4657.616318] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 09:25:01 (1786195501) === [ 4658.744561] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 09:25:02 (1786195502) === [ 4671.805676] Lustre: DEBUG MARKER: oleg334-server.virtnet: executing unload_modules_local [ 4672.831270] Key type lgssc unregistered [ 4672.961376] LNet: 243427:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4672.965702] LNetError: 243427:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4672.976265] LNet: Removed LNI 192.168.203.134@tcp [ 4673.304428] Key type .llcrypt unregistered [ 4673.306461] Key type ._llcrypt unregistered