[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 421531915 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2880904K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002262] x2apic enabled [ 0.003006] Switched APIC routing to physical x2apic. [ 0.004009] kvm-guest: setup PV IPIs [ 0.006445] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.007021] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.008007] pid_max: default: 32768 minimum: 301 [ 0.009133] LSM: Security Framework initializing [ 0.010033] Yama: becoming mindful. [ 0.011024] SELinux: Initializing. [ 0.012046] *** VALIDATE selinux *** [ 0.020720] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024086] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025134] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026087] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027099] *** VALIDATE tmpfs *** [ 0.028486] *** VALIDATE proc *** [ 0.029251] *** VALIDATE cgroup *** [ 0.030006] *** VALIDATE cgroup2 *** [ 0.031287] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.032166] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.033003] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.034025] Spectre V2 : User space: Vulnerable [ 0.035004] Speculative Store Bypass: Vulnerable [ 0.037984] debug: unmapping init [mem 0xffffffffa0259000-0xffffffffa0260fff] [ 0.039783] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.040532] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.041017] ... version: 2 [ 0.041924] ... bit width: 48 [ 0.042007] ... generic registers: 4 [ 0.043007] ... value mask: 0000ffffffffffff [ 0.044006] ... max period: 00007fffffffffff [ 0.045005] ... fixed-purpose events: 3 [ 0.046006] ... event mask: 000000070000000f [ 0.048216] rcu: Hierarchical SRCU implementation. [ 0.050444] smp: Bringing up secondary CPUs ... [ 0.051405] x86: Booting SMP configuration: [ 0.052012] .... node #0, CPUs: #1 #2 #3 [ 0.055013] smp: Brought up 1 node, 4 CPUs [ 0.057010] smpboot: Max logical packages: 1 [ 0.058018] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.241257] node 0 deferred pages initialised in 179ms [ 0.244517] devtmpfs: initialized [ 0.245174] x86/mm: Memory block size: 128MB [ 0.247468] gcov: version magic: 0x41383552 [ 0.249149] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.252060] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.253299] pinctrl core: initialized pinctrl subsystem [ 0.255095] [ 0.255466] ************************************************************* [ 0.257007] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.258005] ** ** [ 0.260006] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.261005] ** ** [ 0.263006] ** This means that this kernel is built to expose internal ** [ 0.265028] ** IOMMU data structures, which may compromise security on ** [ 0.267014] ** your system. ** [ 0.269008] ** ** [ 0.271007] ** If you see this message and you are not debugging the ** [ 0.273008] ** kernel, report this immediately to your vendor! ** [ 0.275007] ** ** [ 0.276006] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.279008] ************************************************************* [ 0.280656] NET: Registered protocol family 16 [ 0.282408] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.285057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.286053] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.289031] cpuidle: using governor menu [ 0.290498] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.291491] PCI: Using configuration type 1 for base access [ 0.293121] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.299082] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.300019] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.301135] cryptd: max_cpu_qlen set to 1000 [ 0.302201] ACPI: Added _OSI(Module Device) [ 0.303010] ACPI: Added _OSI(Processor Device) [ 0.303807] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.305006] ACPI: Added _OSI(Processor Aggregator Device) [ 0.307920] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.312394] ACPI: Interpreter enabled [ 0.313030] ACPI: PM: (supports S0 S3 S4 S5) [ 0.313852] ACPI: Using IOAPIC for interrupt routing [ 0.315051] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.317346] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.324119] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.325016] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.327008] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.329061] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.331929] acpiphp: Slot [2] registered [ 0.333066] acpiphp: Slot [5] registered [ 0.333908] acpiphp: Slot [6] registered [ 0.335084] acpiphp: Slot [7] registered [ 0.336061] acpiphp: Slot [8] registered [ 0.337091] acpiphp: Slot [9] registered [ 0.338053] acpiphp: Slot [10] registered [ 0.338960] acpiphp: Slot [3] registered [ 0.340069] acpiphp: Slot [4] registered [ 0.342074] acpiphp: Slot [11] registered [ 0.343111] acpiphp: Slot [12] registered [ 0.345113] acpiphp: Slot [13] registered [ 0.346062] acpiphp: Slot [14] registered [ 0.346928] acpiphp: Slot [15] registered [ 0.348058] acpiphp: Slot [16] registered [ 0.349098] acpiphp: Slot [17] registered [ 0.351052] acpiphp: Slot [18] registered [ 0.352082] acpiphp: Slot [19] registered [ 0.353073] acpiphp: Slot [20] registered [ 0.355061] acpiphp: Slot [21] registered [ 0.356059] acpiphp: Slot [22] registered [ 0.358048] acpiphp: Slot [23] registered [ 0.359058] acpiphp: Slot [24] registered [ 0.361063] acpiphp: Slot [25] registered [ 0.362089] acpiphp: Slot [26] registered [ 0.363051] acpiphp: Slot [27] registered [ 0.365090] acpiphp: Slot [28] registered [ 0.366064] acpiphp: Slot [29] registered [ 0.368053] acpiphp: Slot [30] registered [ 0.370063] acpiphp: Slot [31] registered [ 0.371073] PCI host bridge to bus 0000:00 [ 0.373012] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.375009] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.377010] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.380011] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.383012] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.386011] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.388160] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.392104] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.395322] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.400971] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.403466] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.405006] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.407007] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.408006] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.409378] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.410434] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.412017] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.413395] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.416005] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.421905] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.425010] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.428825] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.431782] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.437009] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.449014] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.457746] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.464012] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.470033] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.484015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.492548] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.497798] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.501852] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.515023] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.523132] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.528011] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.534016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.546018] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.552526] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.555828] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.559851] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.574012] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.582494] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.585904] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.588864] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.599940] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.607760] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.609185] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.610162] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.611186] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.612185] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.617608] iommu: Default domain type: Passthrough [ 0.618371] SCSI subsystem initialized [ 0.619100] ACPI: bus type USB registered [ 0.620059] usbcore: registered new interface driver usbfs [ 0.621115] usbcore: registered new interface driver hub [ 0.622058] usbcore: registered new device driver usb [ 0.623125] pps_core: LinuxPPS API ver. 1 registered [ 0.625006] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.627042] PTP clock support registered [ 0.629099] EDAC MC: Ver: 3.0.0 [ 0.630129] PCI: Using ACPI for IRQ routing [ 0.631593] NetLabel: Initializing [ 0.632007] NetLabel: domain hash size = 128 [ 0.633016] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.634088] NetLabel: unlabeled traffic allowed by default [ 0.636174] vgaarb: loaded [ 0.637293] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.639009] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.645466] clocksource: Switched to clocksource kvm-clock [ 0.746150] VFS: Disk quotas dquot_6.6.0 [ 0.747541] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.749135] *** VALIDATE ramfs *** [ 0.750275] *** VALIDATE hugetlbfs *** [ 0.751574] pnp: PnP ACPI init [ 0.753629] pnp: PnP ACPI: found 6 devices [ 0.766715] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.769105] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.770871] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.772657] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.774693] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.776648] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.778917] NET: Registered protocol family 2 [ 0.781036] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.784778] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.787947] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.792824] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.795911] TCP: Hash tables configured (established 65536 bind 65536) [ 0.798519] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.801094] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.803633] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.806286] NET: Registered protocol family 1 [ 0.808446] RPC: Registered named UNIX socket transport module. [ 0.810133] RPC: Registered udp transport module. [ 0.811577] RPC: Registered tcp transport module. [ 0.813073] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.814946] NET: Registered protocol family 44 [ 0.816379] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.818122] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.819904] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.821993] PCI: CLS 0 bytes, default 64 [ 0.823385] Unpacking initramfs... [ 2.189159] debug: unmapping init [mem 0xffffa024fcc54000-0xffffa024fffbffff] [ 2.193080] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.195180] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.197379] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 2.868718] Initialise system trusted keyrings [ 2.871932] Key type blacklist registered [ 2.875514] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.895015] zbud: loaded [ 2.901000] *** VALIDATE nfs *** [ 2.905818] *** VALIDATE nfs4 *** [ 2.908284] pstore: using deflate compression [ 2.913650] Platform Keyring initialized [ 3.046575] NET: Registered protocol family 38 [ 3.047781] Key type asymmetric registered [ 3.049244] Asymmetric key parser 'x509' registered [ 3.051186] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.055213] io scheduler mq-deadline registered [ 3.059271] io scheduler kyber registered [ 3.061956] io scheduler bfq registered [ 3.063871] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.067263] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.070333] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.092680] ACPI: Power Button [PWRF] [ 3.099292] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.105418] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.124610] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.131260] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.154225] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.184180] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.212977] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.217970] Non-volatile memory driver v1.3 [ 3.220446] Linux agpgart interface v0.103 [ 3.262269] virtio_blk virtio1: [vda] 146504 512-byte logical blocks (75.0 MB/71.5 MiB) [ 3.265505] vda: detected capacity change from 0 to 75010048 [ 3.283330] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.286941] vdb: detected capacity change from 0 to 1073741824 [ 3.303659] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.306752] vdc: detected capacity change from 0 to 2621440000 [ 3.321607] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.324848] vdd: detected capacity change from 0 to 2621440000 [ 3.344351] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.347652] vde: detected capacity change from 0 to 4294967296 [ 3.367875] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.371922] vdf: detected capacity change from 0 to 4294967296 [ 3.381413] libphy: Fixed MDIO Bus: probed [ 3.394467] usbcore: registered new interface driver usbserial_generic [ 3.401886] usbserial: USB Serial support registered for generic [ 3.408385] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.415820] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.418825] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.422601] mousedev: PS/2 mouse device common for all mice [ 3.427773] rtc_cmos 00:05: RTC can wake from S4 [ 3.432253] rtc_cmos 00:05: registered as rtc0 [ 3.434848] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.442171] intel_pstate: CPU model not supported [ 3.450389] hid: raw HID events driver (C) Jiri Kosina [ 3.451725] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.458039] usbcore: registered new interface driver usbhid [ 3.462308] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.462949] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.480274] usbhid: USB HID core driver [ 3.481692] drop_monitor: Initializing network drop monitor service [ 3.484367] Initializing XFRM netlink socket [ 3.488301] NET: Registered protocol family 10 [ 3.490715] Segment Routing with IPv6 [ 3.494105] NET: Registered protocol family 17 [ 3.496290] mpls_gso: MPLS GSO support [ 3.505171] RAS: Correctable Errors collector initialized. [ 3.509153] AVX version of gcm_enc/dec engaged. [ 3.512967] AES CTR mode by8 optimization enabled [ 3.638201] sched_clock: Marking stable (3638169513, 0)->(4384602964, -746433451) [ 3.641883] registered taskstats version 1 [ 3.647859] Loading compiled-in X.509 certificates [ 3.650068] zswap: loaded using pool lzo/zbud [ 3.673756] Key type big_key registered [ 3.708236] Key type encrypted registered [ 3.712925] ima: No TPM chip found, activating TPM-bypass! [ 3.717319] ima: Allocated hash algorithm: sha1 [ 3.719361] ima: No architecture policies found [ 3.722954] evm: Initialising EVM extended attributes: [ 3.729526] evm: security.selinux [ 3.731551] evm: security.ima [ 3.734250] evm: security.capability [ 3.737579] evm: HMAC attrs: 0x1 [ 3.743301] rtc_cmos 00:05: setting system clock to 2026-06-28 07:08:02 UTC (1782630482) [ 3.752639] debug: unmapping init [mem 0xffffffffa1203000-0xffffffffa13fffff] [ 3.755089] debug: unmapping init [mem 0xffffffff9ff82000-0xffffffffa0258fff] [ 3.763101] Write protecting the kernel read-only data: 28672k [ 3.766763] debug: unmapping init [mem 0xffffffff9e603000-0xffffffff9e7fffff] [ 3.770318] debug: unmapping init [mem 0xffffffff9ef14000-0xffffffff9effffff] [ 3.820476] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.831421] systemd[1]: Detected virtualization kvm. [ 3.833860] systemd[1]: Detected architecture x86-64. [ 3.836390] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.860510] systemd[1]: No hostname configured. [ 3.862207] systemd[1]: Set hostname to . [ 3.864346] random: systemd: uninitialized urandom read (16 bytes read) [ 3.867172] systemd[1]: Initializing machine ID from random generator. [ 4.083273] random: systemd: uninitialized urandom read (16 bytes read) [ 4.085216] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 4.091828] random: systemd: uninitialized urandom read (16 bytes read) [ 4.094474] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 4.100922] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... [ OK ] Reached target Swap. Starting Setup Virtual Console... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.119738] device-mapper: uevent: version 1.0.3 [ 5.122615] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon[ 7.489145] virtio_net virtio0 ens2: renamed from eth0 . [ 7.715568] random: fast init done Starting dracut initqueue hook... [ 8.427705] scsi host0: ata_piix [ 9.056554] scsi host1: ata_piix [ 9.060244] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 9.070618] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 14.378740] random: crng init done [ 14.385745] random: 7 urandom warning(s) missed due to ratelimiting [ 16.668872] dracut-initqueue[584]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 18.388234] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 20.451462] printk: systemd: 26 output lines suppressed due to ratelimiting [ 21.534463] SELinux: Disabled at runtime. [ 21.699722] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 21.709839] systemd[1]: Detected virtualization kvm. [ 21.711959] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 22.965821] systemd[1]: initrd-switch-root.service: Succeeded. [ 22.969669] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 22.977604] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 22.983313] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 22.990661] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 23.005170] systemd[1]: Starting Journal Service... Starting Journal Service... [ 23.016770] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... [ OK ] Stopped target Switch Root. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Kernel Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target rpc_pipefs.target. [ 23.122587] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-serial\x2dgetty.slice. Starting Apply Kernel Variables... [ OK ] Stopped target Initrd Root File System. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting Huge Pages File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Control Socket. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Initrd File Systems. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting udev Coldplug all Devices... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 25.482191] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 26.427960] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 26.630821] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 26.902078] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 26.963124] EDAC sbridge: Ver: 1.1.2 [ 30.551862] Key type dns_resolver registered [ 30.935767] NFS: Registering the id_resolver key type [ 30.937975] Key type id_resolver registered [ 30.939970] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Login Service. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg301-server login: [ 59.338696] libcfs: loading out-of-tree module taints kernel. [ 59.356818] Key type ._llcrypt registered [ 59.357945] Key type .llcrypt registered [ 59.394085] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [ 66.401236] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 67.060859] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 67.067037] alg: No test for adler32 (adler32-zlib) [ 68.031213] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 68.365916] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 69.991128] Key type lgssc registered [ 70.503701] Lustre: Echo OBD driver; http://www.lustre.org/ [ 76.485659] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 91.665108] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 96.348240] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 96.356068] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 97.490131] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 97.502054] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 97.539695] Lustre: lustre-MDT0000: new disk, initializing [ 97.570619] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 97.579238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 98.906302] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 103.990516] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 104.028894] Lustre: 6476:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 104.040357] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 104.042206] Lustre: Skipped 1 previous similar message [ 104.077554] Lustre: lustre-MDT0001: new disk, initializing [ 104.099768] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 104.108642] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 104.112164] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 105.451855] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 107.758781] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 110.925185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 111.046798] Lustre: lustre-OST0000: new disk, initializing [ 111.048463] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 111.050401] Lustre: 8416:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 111.066627] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 112.791635] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 113.450294] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 113.452903] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 113.467865] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 118.125594] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 118.171775] Lustre: lustre-OST0001: new disk, initializing [ 118.174087] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 118.177128] Lustre: 9489:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 118.195533] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 118.762667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 118.765834] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 118.780064] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 120.197332] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 126.249479] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 131.917715] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 134.123927] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing check_logdir /tmp/testlogs/ [ 135.988710] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing yml_node [ 137.678607] Lustre: DEBUG MARKER: Client: 2.17.54.83 [ 138.572477] Lustre: DEBUG MARKER: MDS: 2.17.54.83 [ 139.475076] Lustre: DEBUG MARKER: OSS: 2.17.54.83 [ 140.044871] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Jun 28 03:10:18 EDT 2026 [ 146.297684] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 146.878770] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 159.711812] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 159.713294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 159.717852] Lustre: Skipped 3 previous similar messages [ 159.721338] Lustre: Skipped 1 previous similar message [ 164.834591] Lustre: server umount lustre-MDT0000 complete [ 166.263686] LustreError: 7426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782630645 with bad export cookie 16825823740965928028 [ 166.265184] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 166.268995] LustreError: 7426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 169.952257] LustreError: 12115:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 169.953221] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 169.953594] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 169.965509] LustreError: 12115:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 169.970467] Lustre: Skipped 1 previous similar message [ 172.001110] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 172.001110] LustreError: 6484:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 172.001125] Lustre: Skipped 1 previous similar message [ 172.004106] LustreError: 6484:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 172.510162] Lustre: server umount lustre-MDT0001 complete [ 188.383118] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 188.425661] Lustre: server umount lustre-OST0000 complete [ 204.255119] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 204.357231] Lustre: server umount lustre-OST0001 complete [ 206.820260] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [ 210.340233] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 223.301103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 223.436816] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 223.449936] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 223.488258] Lustre: lustre-MDT0000: new disk, initializing [ 223.514204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 223.522448] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 225.010403] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 229.666484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 229.717319] Lustre: 15031:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 229.845647] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 231.513764] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 236.015902] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 238.463713] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 241.307035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 241.489069] Lustre: lustre-OST0000: new disk, initializing [ 241.491147] Lustre: Skipped 1 previous similar message [ 241.493359] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 241.496127] Lustre: Skipped 2 previous similar messages [ 241.498765] Lustre: 16999:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 241.542883] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 241.547527] Lustre: Skipped 1 previous similar message [ 242.757539] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 242.762727] Lustre: Skipped 1 previous similar message [ 242.765280] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 242.820469] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 244.040617] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 248.645635] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 250.423191] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 250.523444] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 252.186357] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 252.275981] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 253.407940] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 253.411763] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 253.416522] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 253.417856] Lustre: Skipped 1 previous similar message [ 253.423030] Lustre: Skipped 1 previous similar message [ 258.534082] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 258.537491] Lustre: Skipped 1 previous similar message [ 259.267653] Lustre: server umount lustre-OST0000 complete [ 263.648703] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 263.653432] Lustre: Skipped 1 previous similar message [ 266.975852] Lustre: server umount lustre-MDT0000 complete [ 268.419304] LustreError: 18335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782630747 with bad export cookie 16825823740965929547 [ 268.422305] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 268.424105] LustreError: 18335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 268.589366] Lustre: server umount lustre-MDT0001 complete [ 269.963461] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 03:12:28 (1782630748) [ 272.191219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 272.394343] LustreError: 19259:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 272.418620] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 273.995370] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 277.245093] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 278.909063] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 281.773569] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 284.013464] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 286.626671] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 288.854518] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 291.726088] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 308.705287] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 308.708428] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 308.712679] LustreError: Skipped 2 previous similar messages [ 308.713411] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 308.713415] Lustre: Skipped 2 previous similar messages [ 308.723644] Lustre: Skipped 1 previous similar message [ 311.508879] Lustre: server umount lustre-OST0000 complete [ 313.311871] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 319.212358] Lustre: server umount lustre-MDT0000 complete [ 320.896460] LustreError: 19995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782630799 with bad export cookie 16825823740965930429 [ 320.899859] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 320.904648] LustreError: 19995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 321.115345] Lustre: server umount lustre-MDT0001 complete [ 325.977343] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 327.392456] Key type lgssc unregistered [ 327.554637] LNet: 22810:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 327.560039] LNetError: 22810:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 327.569463] LNet: Removed LNI 192.168.203.101@tcp [ 328.018168] Key type .llcrypt unregistered [ 328.020159] Key type ._llcrypt unregistered [ 339.213892] Key type ._llcrypt registered [ 339.215793] Key type .llcrypt registered [ 339.264940] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 339.646158] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 339.703422] alg: No test for adler32 (adler32-zlib) [ 340.585502] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 340.695953] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 342.311171] Key type lgssc registered [ 342.771304] Lustre: Echo OBD driver; http://www.lustre.org/ [ 346.355304] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 346.364919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 347.634632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 349.342170] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 352.717255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 352.872554] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 354.464943] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 357.532535] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 359.887110] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 362.562682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 362.757514] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 364.988387] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 367.782325] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 373.031652] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 378.337603] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 378.337667] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 378.339592] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 378.347350] Lustre: Skipped 1 previous similar message [ 379.506592] Lustre: server umount lustre-OST0000 complete [ 382.934362] Lustre: server umount lustre-MDT0000 complete [ 383.457174] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 383.457212] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 383.461368] LustreError: 24200:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 383.466451] Lustre: Skipped 1 previous similar message [ 383.474349] LustreError: 24200:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 384.275345] LustreError: 27424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782630863 with bad export cookie 3832847724362081797 [ 384.278385] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 384.280121] LustreError: 27424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 384.436828] Lustre: server umount lustre-MDT0001 complete [ 388.434639] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 389.477778] Key type lgssc unregistered [ 389.606429] LNet: 28173:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 389.609553] LNetError: 28173:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 389.623470] LNet: Removed LNI 192.168.203.101@tcp [ 389.949313] Key type .llcrypt unregistered [ 389.950525] Key type ._llcrypt unregistered [ 398.988576] Key type ._llcrypt registered [ 398.989684] Key type .llcrypt registered [ 399.027265] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 399.410774] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 399.422565] alg: No test for adler32 (adler32-zlib) [ 400.277768] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 400.370764] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 401.951402] Key type lgssc registered [ 402.349574] Lustre: Echo OBD driver; http://www.lustre.org/ [ 405.170228] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 405.177197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 406.371749] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 407.770225] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 410.805621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 410.976677] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 412.617321] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 415.187069] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 417.242913] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 421.853613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 422.060458] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 424.073114] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 426.725383] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 432.298372] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 440.800962] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 440.802109] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 440.802271] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 440.805686] Lustre: Skipped 1 previous similar message [ 441.460860] Lustre: server umount lustre-OST0000 complete [ 444.965859] Lustre: server umount lustre-MDT0000 complete [ 445.920818] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 445.925803] LustreError: 29561:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 445.936214] LustreError: 29561:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 446.358046] LustreError: 29542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782630925 with bad export cookie 7267253798898883273 [ 446.362859] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 446.364907] LustreError: 29542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 446.531239] Lustre: server umount lustre-MDT0001 complete [ 450.603365] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 451.783107] Key type lgssc unregistered [ 451.922535] LNet: 33453:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 451.926044] LNetError: 33453:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 451.934325] LNet: Removed LNI 192.168.203.101@tcp [ 452.270385] Key type .llcrypt unregistered [ 452.271975] Key type ._llcrypt unregistered [ 461.513767] Key type ._llcrypt registered [ 461.514952] Key type .llcrypt registered [ 461.561417] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 461.943451] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 461.966797] alg: No test for adler32 (adler32-zlib) [ 462.838788] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 462.934090] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 464.519130] Key type lgssc registered [ 464.878659] Lustre: Echo OBD driver; http://www.lustre.org/ [ 467.518522] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 467.526313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 468.727423] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 470.221050] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 473.317143] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 473.449171] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 474.889123] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 477.530033] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 479.486770] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 481.910810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 482.080599] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 484.029736] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 486.575105] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 493.350644] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 503.778563] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 503.778791] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 503.779619] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 503.783714] Lustre: Skipped 1 previous similar message [ 506.113519] Lustre: server umount lustre-OST0000 complete [ 508.896767] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 508.897838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 508.902145] Lustre: Skipped 1 previous similar message [ 508.904991] Lustre: Skipped 1 previous similar message [ 513.693108] Lustre: server umount lustre-MDT0000 complete [ 514.017896] LustreError: 35581:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 514.024749] LustreError: 35581:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 515.053146] LustreError: 34822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782630993 with bad export cookie 9515313749828465195 [ 515.057711] LustreError: 34822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 515.058112] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 515.226961] Lustre: server umount lustre-MDT0001 complete [ 519.132713] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 520.246692] Key type lgssc unregistered [ 520.392483] LNet: 38395:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 520.395699] LNetError: 38395:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 520.404418] LNet: Removed LNI 192.168.203.101@tcp [ 520.763717] Key type .llcrypt unregistered [ 520.765449] Key type ._llcrypt unregistered [ 529.730508] Key type ._llcrypt registered [ 529.731827] Key type .llcrypt registered [ 529.771433] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 530.174729] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 530.188584] alg: No test for adler32 (adler32-zlib) [ 531.050586] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 531.136113] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 532.719133] Key type lgssc registered [ 533.067790] Lustre: Echo OBD driver; http://www.lustre.org/ [ 535.541206] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 535.547500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 536.740278] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 538.006444] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 540.827450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 540.956095] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 542.416850] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 544.899671] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 546.727842] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 549.064917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 549.239756] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 551.317891] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 553.861817] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 562.534388] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 566.752412] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 566.755288] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 566.760375] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 567.776368] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 567.776741] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 567.777324] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 569.639231] Lustre: lustre-OST0000: Not available for connect from 192.168.203.1@tcp (stopping) [ 569.642050] Lustre: Skipped 1 previous similar message [ 572.896523] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 572.899258] Lustre: Skipped 1 previous similar message [ 578.017149] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 578.019660] Lustre: Skipped 2 previous similar messages [ 581.087120] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 581.181526] Lustre: server umount lustre-OST0000 complete [ 594.223321] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 594.224954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 594.228755] Lustre: Skipped 1 previous similar message [ 594.234688] Lustre: Skipped 2 previous similar messages [ 600.407878] Lustre: server umount lustre-MDT0000 complete [ 601.705449] LustreError: 39763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782631080 with bad export cookie 280924011905860827 [ 601.708864] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 601.709595] LustreError: 39763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 607.960027] Lustre: server umount lustre-MDT0001 complete [ 611.574383] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 612.625790] Key type lgssc unregistered [ 612.751403] LNet: 43509:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 612.754584] LNetError: 43509:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 612.761455] LNet: Removed LNI 192.168.203.101@tcp [ 613.074339] Key type .llcrypt unregistered [ 613.075697] Key type ._llcrypt unregistered [ 622.000093] Key type ._llcrypt registered [ 622.001078] Key type .llcrypt registered [ 622.037977] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 622.332147] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 622.338981] alg: No test for adler32 (adler32-zlib) [ 623.210791] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 623.318883] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 624.911297] Key type lgssc registered [ 625.316194] Lustre: Echo OBD driver; http://www.lustre.org/ [ 628.133828] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 628.142214] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 629.337239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 630.770388] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 633.794179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 633.924254] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 635.305756] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 637.857169] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 639.728834] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 642.112774] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 642.276114] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 644.407625] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 647.029116] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 654.565333] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 662.823516] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.1@tcp (stopping) [ 664.544523] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 664.547841] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 664.553303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 665.061234] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 665.066546] Lustre: Skipped 1 previous similar message [ 666.204141] Lustre: server umount lustre-MDT0000 complete [ 667.518467] LustreError: 44879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782631146 with bad export cookie 14583193584874724415 [ 667.522208] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 667.523830] LustreError: 44879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 667.934229] LustreError: 44892:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 667.943104] LustreError: 44892:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 667.944236] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.1@tcp (stopping) [ 667.952210] Lustre: Skipped 3 previous similar messages [ 670.176125] LustreError: 46892:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 670.176232] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 670.188257] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 672.224226] LustreError: 44891:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 673.759357] Lustre: server umount lustre-MDT0001 complete [ 691.775213] Lustre: server umount lustre-OST0000 complete [ 697.416790] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 698.559305] Key type lgssc unregistered [ 698.694055] LNet: 48776:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 698.697657] LNetError: 48776:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 698.707312] LNet: Removed LNI 192.168.203.101@tcp [ 699.054495] Key type .llcrypt unregistered [ 699.056271] Key type ._llcrypt unregistered [ 709.668128] Key type ._llcrypt registered [ 709.669694] Key type .llcrypt registered [ 709.713197] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 710.043300] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 710.055771] alg: No test for adler32 (adler32-zlib) [ 710.919651] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 711.018194] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 712.607153] Key type lgssc registered [ 713.046047] Lustre: Echo OBD driver; http://www.lustre.org/ [ 715.930364] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 715.937164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 731.423395] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 736.543292] LustreError: 50139:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.101@tcp: failed processing log, type 4: rc = -110 [ 763.167185] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 765.094782] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 767.515515] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 929.956449] Lustre: server umount lustre-OST0000 complete [ 935.243639] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 936.225498] Key type lgssc unregistered [ 936.356348] LNet: 51812:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 936.358897] LNetError: 51812:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 936.372245] LNet: Removed LNI 192.168.203.101@tcp [ 936.687389] Key type .llcrypt unregistered [ 936.688730] Key type ._llcrypt unregistered [ 945.464592] Key type ._llcrypt registered [ 945.465911] Key type .llcrypt registered [ 945.506584] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 946.050032] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 946.061365] alg: No test for adler32 (adler32-zlib) [ 946.921384] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 947.014256] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 948.591154] Key type lgssc registered [ 948.969895] Lustre: Echo OBD driver; http://www.lustre.org/ [ 951.570622] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 951.576767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 952.760282] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 954.120614] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 956.886602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 957.019104] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 958.278911] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 960.606700] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 962.331731] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 964.636772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 964.781895] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 966.637307] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 969.066983] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 972.256575] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 972.259649] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 972.265194] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 975.983501] Lustre: server umount lustre-OST0000 complete [ 977.376291] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 977.376338] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 977.377017] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 977.384394] Lustre: Skipped 1 previous similar message [ 982.495877] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 982.497866] Lustre: Skipped 2 previous similar messages [ 983.314433] Lustre: server umount lustre-MDT0000 complete [ 984.422425] LustreError: 53182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782631463 with bad export cookie 17978048730817704865 [ 984.425467] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 984.426089] LustreError: 53182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 984.572663] Lustre: server umount lustre-MDT0001 complete [ 988.129625] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 989.021577] Key type lgssc unregistered [ 989.137349] LNet: 56730:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 989.140287] LNetError: 56730:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 989.152289] LNet: Removed LNI 192.168.203.101@tcp [ 989.437648] Key type .llcrypt unregistered [ 989.439025] Key type ._llcrypt unregistered [ 997.873146] Key type ._llcrypt registered [ 997.874610] Key type .llcrypt registered [ 997.909664] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 998.193377] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 998.200184] alg: No test for adler32 (adler32-zlib) [ 999.040211] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 999.119973] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 1000.695152] Key type lgssc registered [ 1001.012354] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1003.381842] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1003.387386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1018.847348] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1023.967221] LustreError: 58110:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.101@tcp: failed processing log, type 4: rc = -110 [ 1050.591205] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1052.548622] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1054.924776] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1057.168402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1057.390193] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1057.408730] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1058.602104] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1061.256543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1061.371296] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1062.549053] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1064.654230] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1066.265542] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1067.999736] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1068.002472] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1068.006666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1070.562097] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1070.563262] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1075.681202] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1075.683536] Lustre: Skipped 2 previous similar messages [ 1080.801384] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1080.803453] Lustre: Skipped 1 previous similar message [ 1081.311093] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1081.398492] Lustre: server umount lustre-OST0000 complete [ 1095.306271] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1095.310740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1095.312614] Lustre: Skipped 1 previous similar message [ 1101.524912] Lustre: server umount lustre-MDT0000 complete [ 1102.666258] LustreError: 58116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782631581 with bad export cookie 4414949941070564192 [ 1102.668849] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1102.669991] LustreError: 58116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1108.868623] Lustre: server umount lustre-MDT0001 complete [ 1112.052802] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 1112.970893] Key type lgssc unregistered [ 1113.088297] LNet: 61849:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1113.090275] LNetError: 61849:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1113.097272] LNet: Removed LNI 192.168.203.101@tcp [ 1113.385564] Key type .llcrypt unregistered [ 1113.386546] Key type ._llcrypt unregistered [ 1121.440951] Key type ._llcrypt registered [ 1121.441939] Key type .llcrypt registered [ 1121.473481] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 1121.758195] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1121.801052] alg: No test for adler32 (adler32-zlib) [ 1122.656542] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 1122.742856] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 1124.327124] Key type lgssc registered [ 1124.694379] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1127.118563] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1127.124240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1128.301086] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1129.541586] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1132.138500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1132.246328] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1133.467030] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1135.627079] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1137.249293] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1139.176046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1139.308646] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1141.180475] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1143.608939] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1153.638919] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1157.087983] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1157.090181] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1157.094781] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1159.136851] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1159.138526] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1159.139147] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1162.160504] Lustre: server umount lustre-OST0000 complete [ 1164.256756] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1164.257482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1164.260709] Lustre: Skipped 1 previous similar message [ 1164.263959] Lustre: Skipped 2 previous similar messages [ 1169.376094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1169.378325] Lustre: Skipped 1 previous similar message [ 1169.558634] Lustre: server umount lustre-MDT0000 complete [ 1170.820110] LustreError: 63219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782631649 with bad export cookie 15996082644606141933 [ 1170.822328] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1170.823994] LustreError: 63219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1170.968534] Lustre: server umount lustre-MDT0001 complete [ 1174.755187] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 1175.834467] Key type lgssc unregistered [ 1175.969321] LNet: 66785:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1175.972645] LNetError: 66785:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1175.980262] LNet: Removed LNI 192.168.203.101@tcp [ 1176.306062] Key type .llcrypt unregistered [ 1176.307191] Key type ._llcrypt unregistered [ 1185.731399] Key type ._llcrypt registered [ 1185.733079] Key type .llcrypt registered [ 1185.771855] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 1186.196392] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1186.206208] alg: No test for adler32 (adler32-zlib) [ 1187.064402] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 1187.159096] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 1188.735106] Key type lgssc registered [ 1189.139642] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1192.166860] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 03:27:50 (1782631670) [ 1194.371367] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1194.376892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1195.561372] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1196.895188] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1199.692906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1199.818340] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1201.126805] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1203.528210] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1205.372598] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1207.541810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1207.688299] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1209.565194] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1211.969945] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1221.030852] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1224.583295] Lustre: Failing over lustre-MDT0000 [ 1224.791676] Lustre: server umount lustre-MDT0000 complete [ 1225.695787] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1225.698611] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1225.703872] LustreError: 69193:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1225.712499] LustreError: 69193:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1226.209291] LustreError: 68466:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1226.209578] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1226.215045] LustreError: 68466:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1226.218840] Lustre: Skipped 1 previous similar message [ 1227.556992] LustreError: 68460:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1227.563314] LustreError: 68460:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1227.896068] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1227.938638] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1228.005691] Lustre: *** cfs_fail_loc=135, val=0*** [ 1228.048033] LustreError: 71573:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 1228.050284] LustreError: 71573:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.101@tcp: cfg command failed: rc = -2 [ 1228.052970] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1228.052970] [ 1228.055949] LustreError: MGC192.168.203.101@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1228.061112] LustreError: 71561:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1228.063872] LustreError: 71561:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 1228.066542] LustreError: 71561:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 1228.068382] LustreError: 71561:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 1228.078773] Lustre: server umount lustre-MDT0000 complete [ 1228.080185] LustreError: 71561:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1230.352283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1230.395685] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1230.492190] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1230.518879] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1231.832098] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1232.670314] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1235.945757] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1235.955179] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 1235.972516] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1240.543996] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1240.546549] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1240.553214] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1241.059066] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1241.062032] Lustre: Skipped 1 previous similar message [ 1243.121498] Lustre: server umount lustre-OST0000 complete [ 1246.177204] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1246.177630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1246.180657] Lustre: Skipped 2 previous similar messages [ 1246.180862] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1246.182254] Lustre: Skipped 1 previous similar message [ 1246.188730] LustreError: Skipped 1 previous similar message [ 1250.450940] Lustre: server umount lustre-MDT0000 complete [ 1251.296152] LustreError: 69193:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1251.301452] LustreError: 69193:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1251.655990] LustreError: 68446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782631730 with bad export cookie 16202300633187026227 [ 1251.659219] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1251.660655] LustreError: 68446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1251.819329] Lustre: server umount lustre-MDT0001 complete [ 1255.467124] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 1256.534153] Key type lgssc unregistered [ 1256.662418] LNet: 73347:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1256.665571] LNetError: 73347:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1256.672330] LNet: Removed LNI 192.168.203.101@tcp [ 1257.003654] Key type .llcrypt unregistered [ 1257.005421] Key type ._llcrypt unregistered [ 1265.689795] Key type ._llcrypt registered [ 1265.691010] Key type .llcrypt registered [ 1265.726228] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 1266.087793] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1266.115200] alg: No test for adler32 (adler32-zlib) [ 1266.961021] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 1267.049444] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 1268.631156] Key type lgssc registered [ 1268.982845] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1271.471940] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1271.478749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1272.669424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1273.962379] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1276.684389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1276.808782] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1278.087995] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1280.367096] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1282.109925] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1284.257623] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1284.406523] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1286.322967] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1288.667368] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1297.701887] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 1301.361055] Lustre: Failing over lustre-MDT0000 [ 1301.571728] Lustre: server umount lustre-MDT0000 complete [ 1302.495740] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1302.498293] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1302.502608] LustreError: 74735:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1302.507309] LustreError: 74735:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1303.008613] LustreError: 74734:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1303.009431] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1303.014550] LustreError: 74734:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1303.018343] Lustre: Skipped 1 previous similar message [ 1304.357244] LustreError: 74731:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1304.362852] LustreError: 74731:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1304.552242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1304.593488] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1304.655043] Lustre: *** cfs_fail_loc=a05, val=0*** [ 1304.704373] LustreError: 77842:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 1304.707371] LustreError: 77842:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.101@tcp: cfg command failed: rc = -9 [ 1304.710989] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1304.710989] [ 1304.714210] LustreError: MGC192.168.203.101@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1304.718943] LustreError: 77830:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 1304.721727] LustreError: 77830:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 1304.724794] LustreError: 77830:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 1304.727102] LustreError: 77830:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 1304.739222] Lustre: server umount lustre-MDT0000 complete [ 1304.740774] LustreError: 77830:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 1306.808626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1306.846231] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1306.937980] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1306.961684] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1308.193317] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1309.470997] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1312.235348] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1312.243346] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 1312.259824] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 1317.346412] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1317.346622] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1317.348731] LustreError: Skipped 1 previous similar message [ 1317.352990] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1317.355592] Lustre: Skipped 1 previous similar message [ 1319.342322] Lustre: server umount lustre-OST0000 complete [ 1322.464734] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1322.464749] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1322.465966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1322.473616] Lustre: Skipped 2 previous similar messages [ 1326.682310] Lustre: server umount lustre-MDT0000 complete [ 1327.584265] LustreError: 74734:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1327.590257] LustreError: 74734:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1327.882055] LustreError: 74715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782631806 with bad export cookie 10102143350238764624 [ 1327.885300] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1327.885781] LustreError: 74715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1328.039371] Lustre: server umount lustre-MDT0001 complete [ 1331.599390] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 1332.628073] Key type lgssc unregistered [ 1332.772410] LNet: 79616:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1332.775038] LNetError: 79616:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1332.785312] LNet: Removed LNI 192.168.203.101@tcp [ 1333.109976] Key type .llcrypt unregistered [ 1333.111311] Key type ._llcrypt unregistered [ 1337.473568] Key type ._llcrypt registered [ 1337.475395] Key type .llcrypt registered [ 1337.516138] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 1337.622406] Key type .llcrypt unregistered [ 1337.623677] Key type ._llcrypt unregistered [ 1345.677777] Key type ._llcrypt registered [ 1345.679011] Key type .llcrypt registered [ 1345.710185] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 1345.971456] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1345.982550] alg: No test for adler32 (adler32-zlib) [ 1346.826451] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 1346.908327] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 1348.487097] Key type lgssc registered [ 1348.807680] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1351.262156] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1351.267856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1352.453466] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1353.707317] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1356.351872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1356.466734] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1357.657635] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1359.749856] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1361.293773] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1363.339107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1363.477749] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1365.199089] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1367.331911] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1377.829228] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 1381.855770] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1381.859435] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1381.863448] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1382.880726] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1382.881654] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1382.882078] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1386.878554] Lustre: server umount lustre-OST0000 complete [ 1392.607750] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1392.610121] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1392.614556] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1392.616880] Lustre: Skipped 1 previous similar message [ 1394.263231] Lustre: server umount lustre-MDT0000 complete [ 1395.525043] LustreError: 81475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782631874 with bad export cookie 6742105276687139415 [ 1395.528377] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1395.529252] LustreError: 81475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1395.681933] Lustre: server umount lustre-MDT0001 complete [ 1399.191662] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 1400.282645] Key type lgssc unregistered [ 1400.413302] LNet: 85044:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1400.415726] LNetError: 85044:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1400.423246] LNet: Removed LNI 192.168.203.101@tcp [ 1400.725982] Key type .llcrypt unregistered [ 1400.727102] Key type ._llcrypt unregistered [ 1409.320400] Key type ._llcrypt registered [ 1409.321501] Key type .llcrypt registered [ 1409.365582] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 1409.790089] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1409.799751] alg: No test for adler32 (adler32-zlib) [ 1410.653424] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 1410.745802] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 1412.327154] Key type lgssc registered [ 1412.704262] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1415.308106] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1415.314361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1416.503769] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1417.842932] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1420.663087] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1420.797639] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1422.159056] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1424.562260] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1426.361096] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1428.578234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1428.748291] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1430.641078] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1432.957578] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1442.087149] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 1446.369129] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1446.372153] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1446.375939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1447.395328] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1447.395643] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1447.396676] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1452.018758] Lustre: server umount lustre-OST0000 complete [ 1456.607830] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1456.610445] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1456.616660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1456.619401] Lustre: Skipped 1 previous similar message [ 1459.480027] Lustre: server umount lustre-MDT0000 complete [ 1460.734790] LustreError: 86413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782631939 with bad export cookie 2912539641494764651 [ 1460.737846] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1460.738600] LustreError: 86413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1460.907891] Lustre: server umount lustre-MDT0001 complete [ 1464.378593] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 1465.405216] Key type lgssc unregistered [ 1465.544349] LNet: 89981:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1465.547252] LNetError: 89981:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1465.553322] LNet: Removed LNI 192.168.203.101@tcp [ 1465.876928] Key type .llcrypt unregistered [ 1465.878790] Key type ._llcrypt unregistered [ 1474.089057] Key type ._llcrypt registered [ 1474.090116] Key type .llcrypt registered [ 1474.123546] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 1474.478744] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1474.486684] alg: No test for adler32 (adler32-zlib) [ 1475.339131] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 1475.427951] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 1477.007136] Key type lgssc registered [ 1477.366555] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1479.949187] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1479.954946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1481.141746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1482.444342] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1485.226744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1485.351786] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1486.614511] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1488.973572] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1490.805479] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1493.054920] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1493.215173] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1495.174957] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1497.580692] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1506.534665] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 1511.905718] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1511.906312] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1511.910615] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1511.915835] Lustre: Skipped 1 previous similar message [ 1517.026132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1517.028141] Lustre: Skipped 1 previous similar message [ 1517.298024] Lustre: server umount lustre-OST0000 complete [ 1521.119815] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1521.122423] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1521.126105] Lustre: Skipped 1 previous similar message [ 1521.128043] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1522.145145] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1524.629275] Lustre: server umount lustre-MDT0000 complete [ 1525.817996] LustreError: 91350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782632004 with bad export cookie 17656870277821002549 [ 1525.820835] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1525.822853] LustreError: 91350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1525.963639] Lustre: server umount lustre-MDT0001 complete [ 1529.369183] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 1530.358328] Key type lgssc unregistered [ 1530.487408] LNet: 94915:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1530.490298] LNetError: 94915:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1530.498345] LNet: Removed LNI 192.168.203.101@tcp [ 1530.807089] Key type .llcrypt unregistered [ 1530.808589] Key type ._llcrypt unregistered [ 1539.121600] Key type ._llcrypt registered [ 1539.123285] Key type .llcrypt registered [ 1539.159371] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 1539.517793] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1539.529930] alg: No test for adler32 (adler32-zlib) [ 1540.374293] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 1540.453993] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 1542.031123] Key type lgssc registered [ 1542.355433] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1544.687934] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1544.693897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1560.159346] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1565.279305] LustreError: 96278:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.101@tcp: failed processing log, type 4: rc = -110 [ 1591.903175] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1593.710750] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1595.832936] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1597.542122] Lustre: server umount lustre-OST0000 complete [ 1599.952579] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 03:34:38 (1782632078) [ 1601.968586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1602.124340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1603.284587] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1605.944163] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1606.069036] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1607.277906] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1609.441042] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1611.051230] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1613.064953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1613.200841] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1614.899454] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1617.004725] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1627.561265] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 1637.858105] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1637.859237] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1637.859782] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1637.863474] Lustre: Skipped 1 previous similar message [ 1642.038816] Lustre: server umount lustre-OST0000 complete [ 1647.072097] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1647.075211] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1647.080322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1647.082110] Lustre: Skipped 1 previous similar message [ 1648.096774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1648.097480] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1649.375228] Lustre: server umount lustre-MDT0000 complete [ 1650.582267] LustreError: 100585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782632129 with bad export cookie 11050691204264094759 [ 1650.585113] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1650.587712] LustreError: 100585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1650.749460] Lustre: server umount lustre-MDT0001 complete [ 1654.328901] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 1655.362863] Key type lgssc unregistered [ 1655.485399] LNet: 101623:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1655.487706] LNetError: 101623:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1655.497267] LNet: Removed LNI 192.168.203.101@tcp [ 1655.801834] Key type .llcrypt unregistered [ 1655.803037] Key type ._llcrypt unregistered [ 1670.060392] Key type ._llcrypt registered [ 1670.061561] Key type .llcrypt registered [ 1670.099553] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 1670.478159] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1670.551218] alg: No test for adler32 (adler32-zlib) [ 1671.409173] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 1671.500317] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 1673.079136] Key type lgssc registered [ 1673.430354] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1676.009596] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1676.016545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1677.222871] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1678.514148] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1681.268788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1681.395897] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1682.699144] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1685.096927] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1686.940328] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1689.103762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1689.252025] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1691.117279] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1693.560428] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1701.542550] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 1706.978811] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1706.979398] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1706.980943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1706.985125] Lustre: Skipped 1 previous similar message [ 1706.987264] LustreError: Skipped 1 previous similar message [ 1712.097907] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1712.100161] Lustre: Skipped 2 previous similar messages [ 1717.218452] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1717.220345] Lustre: Skipped 1 previous similar message [ 1721.311167] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1721.404375] Lustre: server umount lustre-OST0000 complete [ 1727.456744] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1727.456765] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1727.465264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1727.467524] Lustre: Skipped 1 previous similar message [ 1728.721855] Lustre: server umount lustre-MDT0000 complete [ 1729.910130] LustreError: 103764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782632208 with bad export cookie 69522883083084101 [ 1729.913483] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1729.914424] LustreError: 103764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1730.060256] Lustre: server umount lustre-MDT0001 complete [ 1733.445464] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 1734.435818] Key type lgssc unregistered [ 1734.551497] LNet: 107333:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1734.556154] LNetError: 107333:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1734.572355] LNet: Removed LNI 192.168.203.101@tcp [ 1734.881689] Key type .llcrypt unregistered [ 1734.883441] Key type ._llcrypt unregistered [ 1742.113384] Key type ._llcrypt registered [ 1742.114442] Key type .llcrypt registered [ 1742.152317] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 1742.484085] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1742.519406] alg: No test for adler32 (adler32-zlib) [ 1743.379356] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 1743.464951] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 1745.039131] Key type lgssc registered [ 1745.379243] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1747.861419] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1747.867993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1763.359282] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1768.479316] LustreError: 108553:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.101@tcp: failed processing log, type 4: rc = -110 [ 1795.103193] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1797.029218] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1799.492100] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1801.724703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1801.864896] LustreError: MGC192.168.203.101@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1801.869891] LustreError: 109576:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1801.874188] LustreError: 109576:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 1801.879237] LustreError: 109576:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 1801.947971] Lustre: server umount lustre-MDT0000 complete [ 1801.949460] LustreError: 109576:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1815.080106] Lustre: server umount lustre-OST0000 complete [ 1818.036728] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [ 1821.197486] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 1832.578539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1832.710577] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1832.725195] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1832.765337] Lustre: lustre-MDT0000: new disk, initializing [ 1832.788201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1832.794404] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1834.072421] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1838.320079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1838.364256] Lustre: 112360:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1838.389355] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1838.391600] Lustre: Skipped 1 previous similar message [ 1838.429234] Lustre: lustre-MDT0001: new disk, initializing [ 1838.460712] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1838.472950] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1838.476547] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1839.816245] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1843.520484] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1845.390834] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1847.625845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1847.782639] Lustre: lustre-OST0000: new disk, initializing [ 1847.784479] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1847.787446] Lustre: 114330:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1847.815054] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1849.668384] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1850.487905] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1850.491860] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1850.523679] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1853.344592] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1854.736799] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1854.806153] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1856.176995] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1856.252641] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1861.091533] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1861.093680] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1861.094389] LustreError: Skipped 1 previous similar message [ 1861.099679] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1861.102012] Lustre: Skipped 1 previous similar message [ 1866.210547] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1866.213377] Lustre: Skipped 1 previous similar message [ 1871.327119] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1871.333257] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1871.335013] Lustre: Skipped 1 previous similar message [ 1871.435427] Lustre: server umount lustre-OST0000 complete [ 1874.400024] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1874.402847] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1874.408264] Lustre: Skipped 1 previous similar message [ 1874.410687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1876.450475] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1881.568066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1881.571200] Lustre: Skipped 3 previous similar messages [ 1887.199098] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1887.267107] Lustre: server umount lustre-MDT0000 complete [ 1888.495580] LustreError: 114329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782632367 with bad export cookie 1221105936213590258 [ 1888.500040] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1888.654426] Lustre: server umount lustre-MDT0001 complete [ 1891.266295] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 03:39:29 (1782632369) [ 1892.315292] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 1897.617763] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [ 1900.880704] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 1912.162801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1912.323238] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1912.363175] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1912.399521] Lustre: lustre-MDT0000: new disk, initializing [ 1912.423321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1912.431367] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1913.835824] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1918.006975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1918.051236] Lustre: 119220:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1918.074195] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1918.076067] Lustre: Skipped 1 previous similar message [ 1918.118946] Lustre: lustre-MDT0001: new disk, initializing [ 1918.180681] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1918.183666] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1919.570327] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1923.237976] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1925.013777] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1927.225332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1927.370380] Lustre: lustre-OST0000: new disk, initializing [ 1927.372976] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1927.377015] Lustre: 121186:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1927.404024] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1927.406246] Lustre: Skipped 1 previous similar message [ 1929.375731] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1930.550447] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1930.554968] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1930.583595] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1932.920650] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1934.247328] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1934.319504] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1935.619246] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1935.686341] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1940.960488] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1940.962500] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1940.963728] LustreError: Skipped 1 previous similar message [ 1940.968640] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1940.970936] Lustre: Skipped 3 previous similar messages [ 1942.444476] Lustre: server umount lustre-OST0000 complete [ 1944.031914] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1946.081202] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1946.087516] Lustre: Skipped 2 previous similar messages [ 1949.780865] Lustre: server umount lustre-MDT0000 complete [ 1950.964384] LustreError: 119212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782632429 with bad export cookie 1221105936213591126 [ 1950.967778] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1950.969123] LustreError: 119212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 1951.123766] Lustre: server umount lustre-MDT0001 complete [ 1953.878464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1954.106612] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1955.509938] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1958.276158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1959.673100] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1961.940025] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1963.652874] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1965.790856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1967.759184] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 1970.080603] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1983.145078] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 1984.479772] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1984.481990] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1984.485666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1984.487795] Lustre: Skipped 3 previous similar messages [ 1986.220522] Lustre: server umount lustre-OST0000 complete [ 1989.524157] Lustre: server umount lustre-MDT0000 complete [ 1990.624339] LustreError: 124132:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1990.629466] LustreError: 124132:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1990.764673] LustreError: 123385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782632469 with bad export cookie 1221105936213591994 [ 1990.770133] LustreError: 123385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1990.770553] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1990.939240] Lustre: server umount lustre-MDT0001 complete [ 1994.361841] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 1995.348757] Key type lgssc unregistered [ 1995.472313] LNet: 127049:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1995.474645] LNetError: 127049:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1995.483354] LNet: Removed LNI 192.168.203.101@tcp [ 1995.771870] Key type .llcrypt unregistered [ 1995.773567] Key type ._llcrypt unregistered [ 2001.598734] Key type ._llcrypt registered [ 2001.600182] Key type .llcrypt registered [ 2001.635599] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [ 2007.614187] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 2008.036357] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2008.043681] alg: No test for adler32 (adler32-zlib) [ 2008.889326] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 2008.980129] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 2010.559144] Key type lgssc registered [ 2010.918630] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2022.009283] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2022.016641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2023.160051] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2023.169935] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2023.210208] Lustre: lustre-MDT0000: new disk, initializing [ 2023.231193] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2023.237049] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2024.536101] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2028.612295] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2028.655790] Lustre: 130173:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2028.681509] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2028.683476] Lustre: Skipped 1 previous similar message [ 2028.719522] Lustre: lustre-MDT0001: new disk, initializing [ 2028.751567] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2028.762842] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2028.767633] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2030.119714] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2033.807951] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2035.572931] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2037.830737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2037.998256] Lustre: lustre-OST0000: new disk, initializing [ 2038.000795] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2038.005122] Lustre: 132142:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2038.038213] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2039.979390] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2043.649160] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2044.974216] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2045.432053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2045.435691] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2045.467989] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2046.055950] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 2047.359555] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2047.428927] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2050.530504] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2050.530866] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2050.532825] LustreError: Skipped 1 previous similar message [ 2050.536905] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2050.539836] Lustre: Skipped 1 previous similar message [ 2054.194301] Lustre: server umount lustre-OST0000 complete [ 2055.648554] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2055.649146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2055.655082] Lustre: Skipped 2 previous similar messages [ 2060.768438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2060.770889] Lustre: Skipped 2 previous similar messages [ 2061.530196] Lustre: server umount lustre-MDT0000 complete [ 2062.755587] LustreError: 130167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782632541 with bad export cookie 17228126390158375786 [ 2062.758262] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2062.760105] LustreError: 130167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2062.917145] Lustre: server umount lustre-MDT0001 complete [ 2065.695982] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 03:42:24 (1782632544) [ 2067.854259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2068.063969] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2069.373045] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2072.093061] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2072.220876] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2073.581123] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2075.989495] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2077.839769] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2081.568797] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2081.573818] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2081.577088] Lustre: Skipped 1 previous similar message [ 2084.631915] Lustre: server umount lustre-MDT0000 complete [ 2085.827122] LustreError: 134537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782632564 with bad export cookie 17228126390158376654 [ 2085.830501] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2085.830640] LustreError: 134537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2085.994478] Lustre: server umount lustre-MDT0001 complete [ 2088.498418] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 03:42:47 (1782632567) [ 2090.607647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2106.079589] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2111.199512] LustreError: 136968:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.101@tcp: failed processing log, type 4: rc = -110 [ 2136.799492] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2138.680306] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2140.987167] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2141.726818] Lustre: server umount lustre-OST0000 complete [ 2144.433679] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 03:43:43 (1782632623) [ 2146.606608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2146.825086] LustreError: 138411:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2146.831582] LustreError: 138411:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2148.251071] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2151.049952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2152.439444] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2154.719525] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2156.464530] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2158.748516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2158.928868] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2158.930808] Lustre: Skipped 2 previous similar messages [ 2160.804652] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2163.126347] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2176.991915] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2176.994055] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2176.997037] Lustre: Skipped 1 previous similar message [ 2176.998746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2181.471619] Lustre: server umount lustre-MDT0000 complete [ 2182.624640] LustreError: 138430:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2182.629414] LustreError: 138430:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2182.749174] LustreError: 138391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782632661 with bad export cookie 17228126390158377305 [ 2182.752166] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2182.752725] LustreError: 138391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2182.900290] Lustre: server umount lustre-MDT0001 complete [ 2193.396235] Lustre: server umount lustre-OST0000 complete [ 2195.982292] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 03:44:34 (1782632674) [ 2198.070269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2198.255113] LustreError: 142080:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2198.277048] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2199.561626] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2202.431221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2203.847966] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2206.055464] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2207.806731] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2209.917954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2211.831306] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2214.077988] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2215.418480] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2223.398080] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 2223.629629] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec [ 2224.942721] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2225.016179] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2228.705668] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2228.705864] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2228.709600] Lustre: Skipped 2 previous similar messages [ 2228.715815] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2228.719023] Lustre: Skipped 4 previous similar messages [ 2231.797315] Lustre: server umount lustre-OST0000 complete [ 2233.311789] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2233.313962] LustreError: Skipped 1 previous similar message [ 2239.126709] Lustre: server umount lustre-MDT0000 complete [ 2240.340186] LustreError: 142063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782632719 with bad export cookie 17228126390158378362 [ 2240.344138] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2240.345968] LustreError: 142063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2240.494484] Lustre: server umount lustre-MDT0001 complete [ 2243.031198] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 03:45:21 (1782632721) [ 2245.173034] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2260.639418] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2265.759492] LustreError: 146289:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.101@tcp: failed processing log, type 4: rc = -110 [ 2291.359288] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2291.361143] Lustre: Skipped 2 previous similar messages [ 2293.201861] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2295.630628] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2297.808080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2297.999250] LustreError: 146314:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2298.038808] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 2299.335848] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2301.981153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2303.391534] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2305.644102] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2307.326862] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2308.634511] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2308.698164] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2309.974018] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2310.038795] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2311.648870] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2311.650467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2311.653926] Lustre: Skipped 4 previous similar messages [ 2311.655903] Lustre: Skipped 4 previous similar messages [ 2316.755629] Lustre: server umount lustre-MDT0000 complete [ 2316.769521] LustreError: 147347:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2316.774858] LustreError: 147347:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2317.953086] LustreError: 146296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782632796 with bad export cookie 17228126390158379370 [ 2317.955799] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2317.958819] LustreError: 146296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2318.109232] Lustre: server umount lustre-MDT0001 complete [ 2329.595148] Lustre: server umount lustre-OST0000 complete [ 2332.203753] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 03:46:50 (1782632810) [ 2334.341393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2349.855770] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2354.975441] LustreError: 150478:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.101@tcp: failed processing log, type 4: rc = -110 [ 2382.414592] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2384.613475] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2386.707854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2386.893993] LustreError: 150503:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2386.936099] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 2388.181906] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2390.882786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2392.212069] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2394.564716] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2396.191938] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2398.253566] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2398.320250] Lustre: lustre-OST0001: new disk, initializing [ 2398.321996] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2398.324879] Lustre: 153345:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2400.139044] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2401.716521] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2401.721310] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2401.748534] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2403.611256] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2404.836245] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2404.898671] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2406.114808] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2406.176465] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2406.882279] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2406.886802] Lustre: Skipped 1 previous similar message [ 2406.888894] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2406.890803] Lustre: Skipped 2 previous similar messages [ 2411.488535] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2421.215125] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2421.246096] Lustre: server umount lustre-OST0000 complete [ 2422.246924] LustreError: 150495:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2422.254023] LustreError: 150495:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2427.362679] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2427.365826] LustreError: Skipped 1 previous similar message [ 2428.608345] Lustre: server umount lustre-OST0001 complete [ 2444.255091] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2444.313655] Lustre: server umount lustre-MDT0000 complete [ 2445.499122] LustreError: 150485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782632924 with bad export cookie 17228126390158380322 [ 2445.502522] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2445.502891] LustreError: 150485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2445.649499] Lustre: server umount lustre-MDT0001 complete [ 2452.820895] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 03:48:51 (1782632931) [ 2453.374402] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 2453.905850] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 2454.402312] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 03:48:53 (1782632933) [ 2456.433554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2456.567565] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2456.577441] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2456.648185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2456.650387] Lustre: Skipped 6 previous similar messages [ 2457.865242] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2460.482477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2460.513459] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2460.522556] Lustre: 156902:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2461.911171] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2464.150321] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2465.857793] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2468.026016] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2468.133622] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2469.943103] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2470.554619] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 2472.548675] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2474.118619] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2474.199754] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2475.829908] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2475.919942] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2481.121631] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2481.121856] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2481.125852] LustreError: Skipped 2 previous similar messages [ 2481.132451] Lustre: Skipped 5 previous similar messages [ 2481.136140] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2481.140294] Lustre: Skipped 14 previous similar messages [ 2482.629319] Lustre: server umount lustre-OST0000 complete [ 2496.669341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2498.702900] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2500.900771] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2500.904577] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2500.904588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2501.070407] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2502.526854] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2502.597263] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2503.882378] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2503.963596] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2505.682196] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9bbec32f6000.ost_server_uuid 50 [ 2506.164480] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9bbec32f6000.ost_server_uuid in FULL state after 0 sec [ 2506.220443] LustreError: 156908:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.203.1@tcp) returned error from blocking AST (req@0000000082a8da27 x1869225642876032 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 0000000095b2ad6e/0xef1699ac5eec566e lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.1@tcp remote: 0x228aba9eacb366c1 expref: 5 pid: 156910 timeout: 2606 lvb_type: 0 lru_score: 0 lru_type: 0 [ 2506.234633] LustreError: lustre-MDT0000: A client on nid 192.168.203.1@tcp was evicted due to a lock blocking callback time out: rc -107 [ 2506.237462] LustreError: 156898:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.203.1@tcp ns: mdt-lustre-MDT0000_UUID lock: 0000000095b2ad6e/0xef1699ac5eec566e lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.1@tcp remote: 0x228aba9eacb366c1 expref: 6 pid: 156910 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 2513.384524] Lustre: server umount lustre-OST0000 complete [ 2520.726090] Lustre: server umount lustre-MDT0000 complete [ 2521.568514] LustreError: 157649:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2521.573265] LustreError: 157649:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2521.952336] LustreError: 156894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782633000 with bad export cookie 17228126390158381505 [ 2521.955986] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2521.957494] LustreError: 156894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2528.157448] Lustre: server umount lustre-MDT0001 complete [ 2531.909858] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 2532.970349] Key type lgssc unregistered [ 2533.101367] LNet: 163003:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2533.104862] LNetError: 163003:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2533.116284] LNet: Removed LNI 192.168.203.101@tcp [ 2533.406053] Key type .llcrypt unregistered [ 2533.407155] Key type ._llcrypt unregistered [ 2541.988908] Key type ._llcrypt registered [ 2541.990591] Key type .llcrypt registered [ 2542.030409] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 2542.368519] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2542.400405] alg: No test for adler32 (adler32-zlib) [ 2543.260379] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 2543.353095] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 2544.935140] Key type lgssc registered [ 2545.294077] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2547.783069] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2547.788569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2548.982241] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2550.223960] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2552.954884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2553.073530] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2554.409324] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2556.691840] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2558.488941] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2560.653724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2560.815488] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2562.725454] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2565.167288] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2575.142131] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 2577.807947] Lustre: Failing over lustre-MDT0000 [ 2578.005950] Lustre: server umount lustre-MDT0000 complete [ 2578.911903] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2578.915466] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2578.921545] LustreError: 164392:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2578.929250] LustreError: 164392:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2580.448386] LustreError: 164410:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2580.450041] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2580.454425] LustreError: 164410:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2580.457929] Lustre: Skipped 1 previous similar message [ 2585.568589] LustreError: 164391:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2585.573356] LustreError: 164391:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2588.904530] LustreError: 164373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.1@tcp arrived at 1782633067 with bad export cookie 9010037275317373263 [ 2590.688512] LustreError: 164392:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2590.693992] LustreError: 164392:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2591.242723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2591.282924] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2591.373473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2591.405178] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2592.631927] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2593.309832] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2593.312219] Lustre: lustre-MDT0000: Denying connection for new client a51231ce-d7c8-4c03-a1f9-dfa003be3477 (at 192.168.203.1@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 2596.839096] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2601.953404] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2601.954790] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2601.956849] Lustre: Skipped 1 previous similar message [ 2601.959974] Lustre: Skipped 1 previous similar message [ 2606.196346] Lustre: server umount lustre-OST0000 complete [ 2607.453215] LustreError: 168138:0:(ldlm_lib.c:2990:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2607.455523] Lustre: 167488:0:(ldlm_lib.c:2390:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2607.459307] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2607.465491] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2607.469245] LustreError: 167488:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000ac01d55a x1869226203714048/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2607.477724] LustreError: 167488:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 2607.480728] LustreError: 167488:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 2607.486872] Lustre: lustre-MDT0000: Recovery over after 0:14, of 2 clients 0 recovered and 2 were evicted. [ 2607.599977] Lustre: server umount lustre-MDT0000 complete [ 2608.840642] LustreError: 164372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782633087 with bad export cookie 9010037275317373592 [ 2608.844134] LustreError: 164372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2608.845328] LustreError: 163702:0:(import.c:709:ptlrpc_connect_import_locked()) can't connect to a closed import [ 2608.845649] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2615.054622] Lustre: server umount lustre-MDT0001 complete [ 2618.586162] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 2619.532527] Key type lgssc unregistered [ 2619.644418] LNet: 168891:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2619.648233] LNetError: 168891:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2619.658269] LNet: Removed LNI 192.168.203.101@tcp [ 2619.950708] Key type .llcrypt unregistered [ 2619.951886] Key type ._llcrypt unregistered [ 2628.317738] Key type ._llcrypt registered [ 2628.318762] Key type .llcrypt registered [ 2628.352147] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 2628.707416] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2628.721747] alg: No test for adler32 (adler32-zlib) [ 2629.573264] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 2629.656794] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 2631.231115] Key type lgssc registered [ 2631.588173] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2634.041400] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2634.047398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2635.233443] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2636.493123] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2639.219893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2639.350842] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2640.582528] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2642.890326] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2644.625143] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2646.891348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2647.057468] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2648.815909] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2651.024405] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2659.366306] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 2664.417790] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2664.418506] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2664.425584] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2664.427325] Lustre: Skipped 1 previous similar message [ 2669.041260] Lustre: server umount lustre-OST0000 complete [ 2674.656423] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2674.657207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2674.660754] Lustre: Skipped 2 previous similar messages [ 2674.663497] Lustre: Skipped 1 previous similar message [ 2676.439301] Lustre: server umount lustre-MDT0000 complete [ 2677.606041] LustreError: 170262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782633156 with bad export cookie 4593658718268246631 [ 2677.609198] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2677.611378] LustreError: 170262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2677.744052] Lustre: server umount lustre-MDT0001 complete [ 2681.190055] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 2682.170864] Key type lgssc unregistered [ 2682.297412] LNet: 173830:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2682.299882] LNetError: 173830:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2682.312296] LNet: Removed LNI 192.168.203.101@tcp [ 2682.618196] Key type .llcrypt unregistered [ 2682.619478] Key type ._llcrypt unregistered [ 2694.171862] Key type ._llcrypt registered [ 2694.172991] Key type .llcrypt registered [ 2694.212627] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 2694.588045] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2694.621221] alg: No test for adler32 (adler32-zlib) [ 2695.473870] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 2695.562781] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 2697.143170] Key type lgssc registered [ 2697.503355] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2699.982452] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2699.988768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2701.181760] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2702.469135] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2705.162684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2705.303831] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2706.571328] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2708.888694] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2710.574659] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2712.763232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2712.919882] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2714.781464] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2717.065306] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2726.250268] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 2731.488868] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2731.490990] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2731.492414] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2731.494411] Lustre: Skipped 1 previous similar message [ 2735.161657] Lustre: server umount lustre-OST0000 complete [ 2736.608076] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2736.612458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2736.615367] Lustre: Skipped 1 previous similar message [ 2741.215888] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2741.218736] LustreError: Skipped 1 previous similar message [ 2741.220335] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2741.225120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2742.484879] Lustre: server umount lustre-MDT0000 complete [ 2743.700107] LustreError: 175486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782633222 with bad export cookie 7196858877434171696 [ 2743.704121] LustreError: 175486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2743.704321] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2743.856718] Lustre: server umount lustre-MDT0001 complete [ 2747.500431] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 2748.540545] Key type lgssc unregistered [ 2748.669358] LNet: 179057:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2748.672269] LNetError: 179057:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2748.681321] LNet: Removed LNI 192.168.203.101@tcp [ 2748.986127] Key type .llcrypt unregistered [ 2748.987276] Key type ._llcrypt unregistered [ 2757.492367] Key type ._llcrypt registered [ 2757.493567] Key type .llcrypt registered [ 2757.537150] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 2757.949674] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2757.979455] alg: No test for adler32 (adler32-zlib) [ 2758.838889] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 2758.927599] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 2760.503131] Key type lgssc registered [ 2760.867569] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2764.461810] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2764.467453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2765.644304] Lustre: *** cfs_fail_loc=135, val=0*** [ 2765.691456] LustreError: 180556:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2765.693733] LustreError: 180556:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.101@tcp: cfg command failed: rc = -2 [ 2765.696873] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2765.696873] [ 2765.701189] LustreError: MGC192.168.203.101@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2765.706488] LustreError: 180518:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2765.709492] LustreError: 180518:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2765.712391] LustreError: 180518:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2765.714617] LustreError: 180518:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2765.787501] Lustre: server umount lustre-MDT0000 complete [ 2765.789082] LustreError: 180518:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2767.906874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2768.097576] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2769.320518] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2772.046489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2772.179319] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2773.505197] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2775.891266] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2777.693236] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2781.472739] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2781.477126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2781.479699] Lustre: Skipped 1 previous similar message [ 2784.533823] Lustre: server umount lustre-MDT0000 complete [ 2785.753339] LustreError: 180909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782633264 with bad export cookie 6885269947957609616 [ 2785.756840] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2785.757294] LustreError: 180909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2785.907395] Lustre: server umount lustre-MDT0001 complete [ 2789.433791] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 2790.422534] Key type lgssc unregistered [ 2790.542360] LNet: 183243:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2790.544811] LNetError: 183243:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2790.556257] LNet: Removed LNI 192.168.203.101@tcp [ 2790.867043] Key type .llcrypt unregistered [ 2790.868604] Key type ._llcrypt unregistered [ 2797.587852] Key type ._llcrypt registered [ 2797.589474] Key type .llcrypt registered [ 2797.624495] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 2797.720167] Key type .llcrypt unregistered [ 2797.721315] Key type ._llcrypt unregistered [ 2804.469672] Key type ._llcrypt registered [ 2804.471010] Key type .llcrypt registered [ 2804.506793] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 2804.960992] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2804.974897] alg: No test for adler32 (adler32-zlib) [ 2805.835703] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 2805.938472] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 2807.527126] Key type lgssc registered [ 2807.913657] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2810.672239] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2810.678439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2826.207417] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2831.327312] LustreError: 185350:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.101@tcp: failed processing log, type 4: rc = -110 [ 2857.951220] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2859.866310] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2862.270229] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2864.506251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2864.759674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2864.778736] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2866.155862] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2868.998051] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2869.125892] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2870.404536] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2872.753995] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2874.460356] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2875.351708] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 2883.553876] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2883.555811] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2883.558893] Lustre: Skipped 1 previous similar message [ 2883.562625] Lustre: Skipped 1 previous similar message [ 2887.481726] Lustre: server umount lustre-OST0000 complete [ 2889.695906] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2889.697948] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2889.701425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2893.792291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2893.792991] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2894.808444] Lustre: server umount lustre-MDT0000 complete [ 2896.037490] LustreError: 185357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782633374 with bad export cookie 7438858464596388569 [ 2896.041148] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2896.042692] LustreError: 185357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2896.204954] Lustre: server umount lustre-MDT0001 complete [ 2899.928984] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 2901.011393] Key type lgssc unregistered [ 2901.148495] LNet: 189309:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2901.153445] LNetError: 189309:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2901.163451] LNet: Removed LNI 192.168.203.101@tcp [ 2901.485166] Key type .llcrypt unregistered [ 2901.486328] Key type ._llcrypt unregistered [ 2910.180406] Key type ._llcrypt registered [ 2910.181623] Key type .llcrypt registered [ 2910.229211] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 2910.600492] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2910.607627] alg: No test for adler32 (adler32-zlib) [ 2911.462906] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 2911.583418] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 2913.167119] Key type lgssc registered [ 2913.508047] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2915.998333] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2916.003648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2917.207220] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2918.513391] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2921.292905] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2921.436223] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2922.732204] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2924.995738] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2926.706341] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2928.783285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2928.933052] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2930.760286] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2933.104850] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2942.247149] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 2944.896352] Lustre: Failing over lustre-MDT0000 [ 2945.004368] Lustre: server umount lustre-MDT0000 complete [ 2947.039929] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2947.042034] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2947.045531] LustreError: 190697:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2947.049327] LustreError: 190697:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2947.553559] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2947.554207] LustreError: 190691:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2947.556973] Lustre: Skipped 1 previous similar message [ 2947.562297] LustreError: 190691:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2948.901187] LustreError: 190691:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2952.673822] LustreError: 193192:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2952.680329] LustreError: 193192:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2957.792224] LustreError: 190697:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2957.799574] LustreError: 190697:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2957.813954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2957.851765] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2957.937860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2957.957306] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2959.134539] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2959.159128] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 2960.590629] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 2963.431889] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2963.439545] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2963.454945] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 2965.051535] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 2971.104929] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2971.107363] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2971.111184] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2973.152248] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2973.155602] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2973.163216] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2975.281668] Lustre: server umount lustre-OST0000 complete [ 2978.785017] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2978.785185] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2978.785739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2978.785743] Lustre: Skipped 1 previous similar message [ 2978.794560] Lustre: Skipped 1 previous similar message [ 2982.610514] Lustre: server umount lustre-MDT0000 complete [ 2983.769650] LustreError: 190677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782633462 with bad export cookie 9049491477288052664 [ 2983.772725] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2983.772823] LustreError: 190677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2983.913888] Lustre: server umount lustre-MDT0001 complete [ 2987.380583] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 2988.467652] Key type lgssc unregistered [ 2988.607301] LNet: 195595:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2988.609620] LNetError: 195595:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2988.621324] LNet: Removed LNI 192.168.203.101@tcp [ 2988.922351] Key type .llcrypt unregistered [ 2988.923455] Key type ._llcrypt unregistered [ 2997.593466] Key type ._llcrypt registered [ 2997.594576] Key type .llcrypt registered [ 2997.637788] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 2998.171426] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2998.180632] alg: No test for adler32 (adler32-zlib) [ 2999.027380] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 2999.116027] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 3000.687146] Key type lgssc registered [ 3001.005821] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3003.341478] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3003.347985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3004.562395] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3005.907877] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3008.543970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3008.662185] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3009.863648] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3012.035604] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3013.666788] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3015.850070] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3016.003839] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3017.787416] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3020.001233] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3029.350759] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 3030.876889] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 3035.351079] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 3043.981375] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3055.076745] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3055.076761] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3055.079082] LustreError: Skipped 1 previous similar message [ 3055.079774] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3055.083074] Lustre: Skipped 1 previous similar message [ 3057.193752] Lustre: server umount lustre-OST0000 complete [ 3060.193059] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3060.193214] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3060.197437] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3060.200051] Lustre: Skipped 1 previous similar message [ 3060.204916] Lustre: Skipped 2 previous similar messages [ 3064.473709] Lustre: server umount lustre-MDT0000 complete [ 3065.311881] LustreError: 196984:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3065.316626] LustreError: 196984:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3065.602406] LustreError: 196963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782633544 with bad export cookie 2899500340467354819 [ 3065.605548] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3065.605738] LustreError: 196963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3065.767961] Lustre: server umount lustre-MDT0001 complete [ 3069.191449] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 3070.051476] Key type lgssc unregistered [ 3070.164312] LNet: 200851:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3070.167245] LNetError: 200851:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3070.177306] LNet: Removed LNI 192.168.203.101@tcp [ 3070.447528] Key type .llcrypt unregistered [ 3070.448582] Key type ._llcrypt unregistered [ 3077.416620] Key type ._llcrypt registered [ 3077.417876] Key type .llcrypt registered [ 3077.459890] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 3077.812300] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3077.842037] alg: No test for adler32 (adler32-zlib) [ 3078.681783] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 3078.761829] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 3080.343135] Key type lgssc registered [ 3080.684976] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3083.204358] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3083.211053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3084.430968] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3085.840263] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3088.621694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3088.744513] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3090.046035] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3092.393824] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3094.134923] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3096.257790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3096.420172] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3098.296288] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3100.531021] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3109.733714] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 3112.574283] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 3123.928125] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 3134.229954] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 3144.148484] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 3154.912159] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3154.916216] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3154.923519] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3155.936879] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3155.936989] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3155.937408] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3155.944016] Lustre: Skipped 1 previous similar message [ 3157.742554] Lustre: server umount lustre-OST0000 complete [ 3160.543737] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3160.545977] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3160.549511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3166.175961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3166.178249] Lustre: Skipped 3 previous similar messages [ 3171.296186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3171.297812] Lustre: Skipped 1 previous similar message [ 3173.343135] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3173.402600] Lustre: server umount lustre-MDT0000 complete [ 3174.513093] LustreError: 204789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782633653 with bad export cookie 11861258466324082030 [ 3174.516321] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3174.516324] LustreError: 204789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3174.680084] Lustre: server umount lustre-MDT0001 complete [ 3177.943869] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 3178.851742] Key type lgssc unregistered [ 3178.979331] LNet: 207850:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3178.981542] LNetError: 207850:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3178.992275] LNet: Removed LNI 192.168.203.101@tcp [ 3179.278104] Key type .llcrypt unregistered [ 3179.279210] Key type ._llcrypt unregistered [ 3187.436657] Key type ._llcrypt registered [ 3187.437804] Key type .llcrypt registered [ 3187.469783] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 3187.746198] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3187.752973] alg: No test for adler32 (adler32-zlib) [ 3188.593214] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 3188.673625] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 3190.247096] Key type lgssc registered [ 3190.599670] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3193.000080] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3193.005888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3194.181256] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3195.455722] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3198.193033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3198.322218] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3199.607112] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3201.967877] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3203.737554] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3205.886668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3206.047292] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3207.924244] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3210.222385] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3218.342328] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 3224.543923] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3224.545970] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3224.549448] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3228.642041] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3228.642898] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3228.643063] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3233.761635] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3233.764275] Lustre: Skipped 2 previous similar messages [ 3238.367118] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3238.459795] Lustre: server umount lustre-OST0000 complete [ 3244.000677] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3244.001538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3244.006133] Lustre: Skipped 1 previous similar message [ 3245.787333] Lustre: server umount lustre-MDT0000 complete [ 3246.971287] LustreError: 209233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782633725 with bad export cookie 9861844087558772399 [ 3246.974351] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3246.976331] LustreError: 209233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3247.124086] Lustre: server umount lustre-MDT0001 complete [ 3250.543723] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 3251.493458] Key type lgssc unregistered [ 3251.609375] LNet: 212953:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3251.611607] LNetError: 212953:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3251.623309] LNet: Removed LNI 192.168.203.101@tcp [ 3251.912672] Key type .llcrypt unregistered [ 3251.913732] Key type ._llcrypt unregistered [ 3258.973483] Key type ._llcrypt registered [ 3258.974516] Key type .llcrypt registered [ 3259.015117] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 3259.508601] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3259.517635] alg: No test for adler32 (adler32-zlib) [ 3260.378492] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 3260.471516] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 3262.055127] Key type lgssc registered [ 3262.397378] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3264.778625] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3264.784041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3265.972141] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3267.216588] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3269.936975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3270.055413] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3271.314405] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3273.612214] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3275.352201] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3277.568623] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3277.736498] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3279.772701] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3282.137211] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3292.069800] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 3297.252255] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3297.253767] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3297.253846] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3297.257139] Lustre: Skipped 1 previous similar message [ 3301.681651] Lustre: server umount lustre-OST0000 complete [ 3305.951924] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3305.955418] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3305.961118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3305.962701] Lustre: Skipped 1 previous similar message [ 3307.488869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3307.489737] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3312.608157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3312.611894] Lustre: Skipped 2 previous similar messages [ 3317.215087] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3317.268632] Lustre: server umount lustre-MDT0000 complete [ 3317.728050] LustreError: 214195:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3317.732413] LustreError: 214195:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3318.371279] LustreError: 216040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782633797 with bad export cookie 8305953520645026193 [ 3318.374227] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3318.374488] LustreError: 216040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3318.507552] Lustre: server umount lustre-MDT0001 complete [ 3321.714043] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 3322.547566] Key type lgssc unregistered [ 3322.657381] LNet: 217896:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3322.660244] LNetError: 217896:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3322.669260] LNet: Removed LNI 192.168.203.101@tcp [ 3322.938240] Key type .llcrypt unregistered [ 3322.939344] Key type ._llcrypt unregistered [ 3329.760900] Key type ._llcrypt registered [ 3329.761942] Key type .llcrypt registered [ 3329.794603] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 3330.100684] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3330.107340] alg: No test for adler32 (adler32-zlib) [ 3330.956947] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 3331.047797] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 3332.623143] Key type lgssc registered [ 3332.954014] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3335.362407] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3335.367803] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3336.542340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3337.745347] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3340.507650] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3340.635917] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3341.876218] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3344.102973] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3345.847607] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3347.880814] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3348.037974] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3349.735193] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3351.885901] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3361.381898] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 3364.950575] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3364.979062] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3365.051571] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3366.832747] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3369.314545] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3372.864745] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec [ 3383.206870] Lustre: Permanently deactivating lustre-OST0001 [ 3383.213241] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3393.529247] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 3399.102354] Lustre: Permanently reactivating lustre-OST0001 [ 3399.107670] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 3399.109627] Lustre: Skipped 2 previous similar messages [ 3401.271655] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3401.275482] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3401.277690] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3401.281165] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3401.292610] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3406.304220] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3406.304377] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3406.306490] LustreError: Skipped 1 previous similar message [ 3406.306897] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3406.310174] Lustre: Skipped 2 previous similar messages [ 3406.314514] Lustre: Skipped 1 previous similar message [ 3410.401823] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3410.403635] Lustre: Skipped 1 previous similar message [ 3412.221544] Lustre: server umount lustre-OST0001 complete [ 3415.523894] LustreError: 220973:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3415.524902] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3415.526627] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3415.531250] LustreError: 220973:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3415.541641] Lustre: Skipped 1 previous similar message [ 3419.571405] Lustre: server umount lustre-OST0000 complete [ 3422.688055] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3422.690453] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3422.694271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3422.696328] Lustre: Skipped 1 previous similar message [ 3426.967820] Lustre: server umount lustre-MDT0000 complete [ 3428.104269] LustreError: 219135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782633906 with bad export cookie 12928548703210975921 [ 3428.107902] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3428.110397] LustreError: 219135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3428.260794] Lustre: server umount lustre-MDT0001 complete [ 3431.699593] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 3432.732338] Key type lgssc unregistered [ 3432.857581] LNet: 224242:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3432.863462] LNetError: 224242:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3432.876562] LNet: Removed LNI 192.168.203.101@tcp [ 3433.282242] Key type .llcrypt unregistered [ 3433.284689] Key type ._llcrypt unregistered [ 3447.424501] Key type ._llcrypt registered [ 3447.425618] Key type .llcrypt registered [ 3447.458676] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 3447.833511] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3447.844932] alg: No test for adler32 (adler32-zlib) [ 3448.684688] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 3448.765798] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 3450.335107] Key type lgssc registered [ 3450.655456] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3453.091950] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3453.098721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3454.232578] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3454.240809] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3454.303162] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3455.547604] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3458.376435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3458.410695] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3458.422520] Lustre: 226421:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3458.516640] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3459.793822] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3462.091921] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3463.837168] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3466.038777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3466.149609] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3466.224065] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3468.140326] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3470.475037] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3472.430301] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 3477.112596] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 3479.534265] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 3489.077979] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 3497.589870] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3506.092149] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 3530.592972] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 3530.595082] Lustre: Skipped 2 previous similar messages [ 3562.864255] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3562.867589] Lustre: Skipped 3 previous similar messages [ 3634.954922] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 3634.960028] Lustre: Skipped 9 previous similar messages [ 3641.315227] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3641.315650] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3641.316689] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3641.321820] Lustre: Skipped 1 previous similar message [ 3641.331225] LustreError: Skipped 1 previous similar message [ 3642.829113] Lustre: server umount lustre-OST0000 complete [ 3646.432647] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3646.433274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3646.436026] Lustre: Skipped 1 previous similar message [ 3646.439116] Lustre: Skipped 2 previous similar messages [ 3651.552186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3651.554311] Lustre: Skipped 1 previous similar message [ 3656.672108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3656.674370] Lustre: Skipped 1 previous similar message [ 3658.207075] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3658.263452] Lustre: server umount lustre-MDT0000 complete [ 3659.389180] LustreError: 227160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782634138 with bad export cookie 5979980304209614972 [ 3659.392252] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3659.392561] LustreError: 227160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3659.530964] Lustre: server umount lustre-MDT0001 complete [ 3662.682371] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 3663.561240] Key type lgssc unregistered [ 3663.691360] LNet: 231200:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3663.693673] LNetError: 231200:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3663.701317] LNet: Removed LNI 192.168.203.101@tcp [ 3664.005583] Key type .llcrypt unregistered [ 3664.007923] Key type ._llcrypt unregistered [ 3672.263075] Key type ._llcrypt registered [ 3672.264168] Key type .llcrypt registered [ 3672.297553] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 3672.628629] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3672.663396] alg: No test for adler32 (adler32-zlib) [ 3673.507879] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 3673.589935] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 3675.167132] Key type lgssc registered [ 3675.487455] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3677.790395] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3677.796206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3678.984605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3680.122696] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3682.734418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3682.859622] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3684.062234] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3686.227882] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3688.011080] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3690.319509] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3690.542468] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3692.555622] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_default_debug -1 all [ 3695.201742] Lustre: DEBUG MARKER: oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3703.846533] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 3718.112549] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3718.118396] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3718.128562] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3719.138348] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3719.139607] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3719.140624] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3721.583305] Lustre: server umount lustre-OST0000 complete [ 3723.744432] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3723.750502] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3723.761094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3723.766135] Lustre: Skipped 1 previous similar message [ 3729.047299] Lustre: server umount lustre-MDT0000 complete [ 3729.375931] LustreError: 234031:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3729.381258] LustreError: 234031:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3730.404663] LustreError: 235187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782634209 with bad export cookie 6467642313468667307 [ 3730.409128] LustreError: MGC192.168.203.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3730.410208] LustreError: 235187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3730.566322] Lustre: server umount lustre-MDT0001 complete [ 3734.171143] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 3735.197118] Key type lgssc unregistered [ 3735.323558] LNet: 236292:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3735.326989] LNetError: 236292:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3735.336382] LNet: Removed LNI 192.168.203.101@tcp [ 3735.660433] Key type .llcrypt unregistered [ 3735.662120] Key type ._llcrypt unregistered [ 3743.660531] Key type ._llcrypt registered [ 3743.661800] Key type .llcrypt registered [ 3743.695484] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 3744.023255] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3744.054809] alg: No test for adler32 (adler32-zlib) [ 3744.898947] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 3744.978872] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 3746.551118] Key type lgssc registered [ 3746.875207] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4013.305430] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 4014.275491] Key type lgssc unregistered [ 4014.401353] LNet: 238290:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4014.403855] LNetError: 238290:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4014.415476] LNet: Removed LNI 192.168.203.101@tcp [ 4014.728636] Key type .llcrypt unregistered [ 4014.730235] Key type ._llcrypt unregistered [ 4022.696423] Key type ._llcrypt registered [ 4022.697521] Key type .llcrypt registered [ 4022.730220] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing set_hostid [ 4028.896931] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing load_modules_local [ 4029.233063] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4029.266089] alg: No test for adler32 (adler32-zlib) [ 4030.116361] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 4030.205294] LNet: Added LNI 192.168.203.101@tcp [8/256/0/180] [ 4031.783143] Key type lgssc registered [ 4032.138870] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4041.544472] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 3901 sec ========== 04:15:20 (1782634520) [ 4042.056086] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 04:15:20 (1782634520) === [ 4043.057411] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 04:15:21 (1782634521) === [ 4055.485568] Lustre: DEBUG MARKER: oleg301-server.virtnet: executing unload_modules_local [ 4056.415835] Key type lgssc unregistered [ 4056.532353] LNet: 243500:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4056.534938] LNetError: 243500:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4056.548274] LNet: Removed LNI 192.168.203.101@tcp [ 4056.823303] Key type .llcrypt unregistered [ 4056.824383] Key type ._llcrypt unregistered