[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 450091040 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002311] x2apic enabled [ 0.003006] Switched APIC routing to physical x2apic. [ 0.004010] kvm-guest: setup PV IPIs [ 0.006748] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007013] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008008] pid_max: default: 32768 minimum: 301 [ 0.010088] LSM: Security Framework initializing [ 0.011000] Yama: becoming mindful. [ 0.011000] SELinux: Initializing. [ 0.011043] *** VALIDATE selinux *** [ 0.017684] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.022292] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.023134] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.024073] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025094] *** VALIDATE tmpfs *** [ 0.027269] *** VALIDATE proc *** [ 0.028000] *** VALIDATE cgroup *** [ 0.028005] *** VALIDATE cgroup2 *** [ 0.029179] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.030134] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.031005] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.032026] Spectre V2 : User space: Vulnerable [ 0.033007] Speculative Store Bypass: Vulnerable [ 0.036567] debug: unmapping init [mem 0xffffffffb6259000-0xffffffffb6260fff] [ 0.038163] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.039607] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.040020] ... version: 2 [ 0.041008] ... bit width: 48 [ 0.042007] ... generic registers: 4 [ 0.043009] ... value mask: 0000ffffffffffff [ 0.044009] ... max period: 00007fffffffffff [ 0.045006] ... fixed-purpose events: 3 [ 0.046007] ... event mask: 000000070000000f [ 0.048180] rcu: Hierarchical SRCU implementation. [ 0.050135] smp: Bringing up secondary CPUs ... [ 0.051471] x86: Booting SMP configuration: [ 0.052018] .... node #0, CPUs: #1 #2 #3 [ 0.057176] smp: Brought up 1 node, 4 CPUs [ 0.059009] smpboot: Max logical packages: 1 [ 0.060010] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.153180] node 0 deferred pages initialised in 91ms [ 0.156138] devtmpfs: initialized [ 0.157232] x86/mm: Memory block size: 128MB [ 0.160013] gcov: version magic: 0x41383552 [ 0.164013] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.167098] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.170450] pinctrl core: initialized pinctrl subsystem [ 0.173211] [ 0.173861] ************************************************************* [ 0.176014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.178010] ** ** [ 0.180012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.182013] ** ** [ 0.184015] ** This means that this kernel is built to expose internal ** [ 0.187011] ** IOMMU data structures, which may compromise security on ** [ 0.188009] ** your system. ** [ 0.190010] ** ** [ 0.192011] ** If you see this message and you are not debugging the ** [ 0.194014] ** kernel, report this immediately to your vendor! ** [ 0.196013] ** ** [ 0.198013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.201012] ************************************************************* [ 0.203702] NET: Registered protocol family 16 [ 0.205381] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.207053] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.211082] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.215022] cpuidle: using governor menu [ 0.217075] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.219503] PCI: Using configuration type 1 for base access [ 0.221143] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.229054] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.230014] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.233195] cryptd: max_cpu_qlen set to 1000 [ 0.236314] ACPI: Added _OSI(Module Device) [ 0.238011] ACPI: Added _OSI(Processor Device) [ 0.240014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.242011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.246337] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.255663] ACPI: Interpreter enabled [ 0.257038] ACPI: PM: (supports S0 S3 S4 S5) [ 0.258011] ACPI: Using IOAPIC for interrupt routing [ 0.260142] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.261349] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.269815] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.272083] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.274016] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.277080] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.282235] acpiphp: Slot [2] registered [ 0.283200] acpiphp: Slot [5] registered [ 0.285106] acpiphp: Slot [6] registered [ 0.286081] acpiphp: Slot [7] registered [ 0.288149] acpiphp: Slot [8] registered [ 0.289101] acpiphp: Slot [9] registered [ 0.291109] acpiphp: Slot [10] registered [ 0.292140] acpiphp: Slot [3] registered [ 0.294095] acpiphp: Slot [4] registered [ 0.295092] acpiphp: Slot [11] registered [ 0.297119] acpiphp: Slot [12] registered [ 0.298128] acpiphp: Slot [13] registered [ 0.299117] acpiphp: Slot [14] registered [ 0.300089] acpiphp: Slot [15] registered [ 0.302122] acpiphp: Slot [16] registered [ 0.304114] acpiphp: Slot [17] registered [ 0.305104] acpiphp: Slot [18] registered [ 0.306129] acpiphp: Slot [19] registered [ 0.308102] acpiphp: Slot [20] registered [ 0.310320] acpiphp: Slot [21] registered [ 0.312125] acpiphp: Slot [22] registered [ 0.314084] acpiphp: Slot [23] registered [ 0.316074] acpiphp: Slot [24] registered [ 0.317074] acpiphp: Slot [25] registered [ 0.318063] acpiphp: Slot [26] registered [ 0.320063] acpiphp: Slot [27] registered [ 0.321085] acpiphp: Slot [28] registered [ 0.323108] acpiphp: Slot [29] registered [ 0.325244] acpiphp: Slot [30] registered [ 0.327085] acpiphp: Slot [31] registered [ 0.328077] PCI host bridge to bus 0000:00 [ 0.330022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.332022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.334016] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.336020] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.339019] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.341023] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.342179] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.345455] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.349121] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.358765] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.364070] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.366021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.369022] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.371019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.375898] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.378079] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.380032] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.383014] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.388015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.399848] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.403016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.408308] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.414013] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.420016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.436021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.445798] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.452019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.459018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.474026] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.483529] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.489012] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.494012] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.510024] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.519501] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.526018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.530017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.547023] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.557977] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.566019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.573019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.591021] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.604777] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.609019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.614018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.630019] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.640838] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.642244] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.644251] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.645190] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.647225] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.651152] iommu: Default domain type: Passthrough [ 0.653563] SCSI subsystem initialized [ 0.655118] ACPI: bus type USB registered [ 0.657091] usbcore: registered new interface driver usbfs [ 0.658066] usbcore: registered new interface driver hub [ 0.660116] usbcore: registered new device driver usb [ 0.662143] pps_core: LinuxPPS API ver. 1 registered [ 0.664013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.666073] PTP clock support registered [ 0.669042] EDAC MC: Ver: 3.0.0 [ 0.671137] PCI: Using ACPI for IRQ routing [ 0.674285] NetLabel: Initializing [ 0.675010] NetLabel: domain hash size = 128 [ 0.676010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.677087] NetLabel: unlabeled traffic allowed by default [ 0.679137] vgaarb: loaded [ 0.681266] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.682013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.689878] clocksource: Switched to clocksource kvm-clock [ 0.792496] VFS: Disk quotas dquot_6.6.0 [ 0.794155] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.796968] *** VALIDATE ramfs *** [ 0.798456] *** VALIDATE hugetlbfs *** [ 0.800694] pnp: PnP ACPI init [ 0.803151] pnp: PnP ACPI: found 6 devices [ 0.819939] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.823640] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.826202] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.828745] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.831448] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.834953] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.838184] NET: Registered protocol family 2 [ 0.841278] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.847452] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.851306] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.857029] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.860702] TCP: Hash tables configured (established 65536 bind 65536) [ 0.863696] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.866967] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.869445] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.872499] NET: Registered protocol family 1 [ 0.874982] RPC: Registered named UNIX socket transport module. [ 0.877152] RPC: Registered udp transport module. [ 0.878398] RPC: Registered tcp transport module. [ 0.879974] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.882055] NET: Registered protocol family 44 [ 0.883212] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.885209] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.886986] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.888907] PCI: CLS 0 bytes, default 64 [ 0.890474] Unpacking initramfs... [ 2.255142] debug: unmapping init [mem 0xffff8fb7fcc54000-0xffff8fb7fffbffff] [ 2.259499] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.262104] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.264952] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.741502] Initialise system trusted keyrings [ 2.742730] Key type blacklist registered [ 2.744413] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.752502] zbud: loaded [ 2.755831] *** VALIDATE nfs *** [ 2.756828] *** VALIDATE nfs4 *** [ 2.765637] pstore: using deflate compression [ 2.768738] Platform Keyring initialized [ 2.875713] NET: Registered protocol family 38 [ 2.877442] Key type asymmetric registered [ 2.878885] Asymmetric key parser 'x509' registered [ 2.880361] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.882992] io scheduler mq-deadline registered [ 2.885703] io scheduler kyber registered [ 2.888269] io scheduler bfq registered [ 2.891377] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.893904] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.896853] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.900279] ACPI: Power Button [PWRF] [ 2.907098] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.912687] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.931984] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.939868] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.965312] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.992394] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.021068] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.027677] Non-volatile memory driver v1.3 [ 3.029496] Linux agpgart interface v0.103 [ 3.057979] virtio_blk virtio1: [vda] 149760 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.060347] vda: detected capacity change from 0 to 76677120 [ 3.075241] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.077853] vdb: detected capacity change from 0 to 1073741824 [ 3.090907] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.093897] vdc: detected capacity change from 0 to 2621440000 [ 3.106810] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.110388] vdd: detected capacity change from 0 to 2621440000 [ 3.123980] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.127037] vde: detected capacity change from 0 to 4294967296 [ 3.143951] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.146705] vdf: detected capacity change from 0 to 4294967296 [ 3.154332] libphy: Fixed MDIO Bus: probed [ 3.161472] usbcore: registered new interface driver usbserial_generic [ 3.166711] usbserial: USB Serial support registered for generic [ 3.175921] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.181463] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.184072] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.188034] mousedev: PS/2 mouse device common for all mice [ 3.192551] rtc_cmos 00:05: RTC can wake from S4 [ 3.196476] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.200278] rtc_cmos 00:05: registered as rtc0 [ 3.204454] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.204460] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.209305] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.210622] intel_pstate: CPU model not supported [ 3.215933] hid: raw HID events driver (C) Jiri Kosina [ 3.218560] usbcore: registered new interface driver usbhid [ 3.220169] usbhid: USB HID core driver [ 3.221894] drop_monitor: Initializing network drop monitor service [ 3.224140] Initializing XFRM netlink socket [ 3.226126] NET: Registered protocol family 10 [ 3.228960] Segment Routing with IPv6 [ 3.230460] NET: Registered protocol family 17 [ 3.231982] mpls_gso: MPLS GSO support [ 3.236313] RAS: Correctable Errors collector initialized. [ 3.238573] AVX version of gcm_enc/dec engaged. [ 3.239963] AES CTR mode by8 optimization enabled [ 3.302592] sched_clock: Marking stable (3302571690, 0)->(4109827144, -807255454) [ 3.305833] registered taskstats version 1 [ 3.307784] Loading compiled-in X.509 certificates [ 3.309271] zswap: loaded using pool lzo/zbud [ 3.331821] Key type big_key registered [ 3.343048] Key type encrypted registered [ 3.344559] ima: No TPM chip found, activating TPM-bypass! [ 3.346972] ima: Allocated hash algorithm: sha1 [ 3.348757] ima: No architecture policies found [ 3.350317] evm: Initialising EVM extended attributes: [ 3.352038] evm: security.selinux [ 3.353106] evm: security.ima [ 3.353982] evm: security.capability [ 3.355499] evm: HMAC attrs: 0x1 [ 3.357836] rtc_cmos 00:05: setting system clock to 2026-09-08 13:30:22 UTC (1788874222) [ 3.365291] debug: unmapping init [mem 0xffffffffb7203000-0xffffffffb73fffff] [ 3.368533] debug: unmapping init [mem 0xffffffffb5f82000-0xffffffffb6258fff] [ 3.377207] Write protecting the kernel read-only data: 28672k [ 3.380027] debug: unmapping init [mem 0xffffffffb4603000-0xffffffffb47fffff] [ 3.382471] debug: unmapping init [mem 0xffffffffb4f14000-0xffffffffb4ffffff] [ 3.416885] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.424697] systemd[1]: Detected virtualization kvm. [ 3.426625] systemd[1]: Detected architecture x86-64. [ 3.428330] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.452301] systemd[1]: No hostname configured. [ 3.453727] systemd[1]: Set hostname to . [ 3.455242] random: systemd: uninitialized urandom read (16 bytes read) [ 3.458053] systemd[1]: Initializing machine ID from random generator. [ 3.501611] random: ln: uninitialized urandom read (6 bytes read) [ 3.582809] random: systemd: uninitialized urandom read (16 bytes read) [ 3.585323] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.589256] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.592849] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. Starting Setup Virtual Console... Starting Journal Service... [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.152884] device-mapper: uevent: version 1.0.3 [ 4.155542] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ 4.661284] random: fast init done [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.834903] virtio_net virtio0 ens2: renamed from eth0 [ 4.925659] scsi host0: ata_piix [ 4.988213] scsi host1: ata_piix [ 4.989624] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.991639] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.161270] dracut-initqueue[587]: RTNETLINK answers: File exists [ 9.860488] random: crng init done [ 9.861632] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.153759] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.449510] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.795616] SELinux: Disabled at runtime. [ 11.857230] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.867412] systemd[1]: Detected virtualization kvm. [ 11.869682] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.587438] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.591802] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.596666] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.602589] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.606408] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.617351] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.627634] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-getty.slice. [ OK ] Created slice User and Session Slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on initctl Compatibility Named Pipe. Activating swap /dev/disk/by-label/SWAP... Mounting Kernel Debug File System... [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target rpc_pipefs.target. Starting Create list of required st…ce nodes for the current kernel... [ 12.823458] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... Mounting POSIX Message Queue File System... [ OK ] Reached target Slices. Mounting Huge Pages File System... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.149787] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.535446] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.577961] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.711243] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.730273] EDAC sbridge: Ver: 1.1.2 [ 15.508916] Key type dns_resolver registered [ 15.805260] NFS: Registering the id_resolver key type [ 15.807280] Key type id_resolver registered [ 15.808880] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg631-server login: [ 44.353209] libcfs: loading out-of-tree module taints kernel. [ 44.375945] Key type ._llcrypt registered [ 44.378151] Key type .llcrypt registered [ 44.430896] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_hostid [ 53.921316] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 54.657246] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 54.667851] alg: No test for adler32 (adler32-zlib) [ 55.731219] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 56.133990] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 57.791401] Key type lgssc registered [ 58.644258] Lustre: Echo OBD driver; http://www.lustre.org/ [ 66.981352] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 84.599364] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 90.523663] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 90.540288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 91.655209] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 91.670202] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 91.715644] Lustre: lustre-MDT0000: new disk, initializing [ 91.761705] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 91.780089] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 94.026378] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 101.328178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 101.378865] Lustre: 6477:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 101.406299] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 101.410347] Lustre: Skipped 1 previous similar message [ 101.460222] Lustre: lustre-MDT0001: new disk, initializing [ 101.494482] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 101.508256] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 101.513950] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 103.457625] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 106.252263] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 110.612116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 110.741852] Lustre: lustre-OST0000: new disk, initializing [ 110.745226] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 110.749960] Lustre: 8413:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 110.788513] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 113.441280] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 114.233256] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 114.237770] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 114.270757] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 120.268206] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 120.323394] Lustre: lustre-OST0001: new disk, initializing [ 120.325604] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 120.329175] Lustre: 9488:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 120.361820] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 120.530400] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 120.535957] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 120.565334] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 123.024989] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 131.128688] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 135.148524] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 137.590836] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing check_logdir /tmp/testlogs/ [ 139.652152] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing yml_node [ 141.629225] Lustre: DEBUG MARKER: Client: 2.17.58.2 [ 142.724380] Lustre: DEBUG MARKER: MDS: 2.17.58.2 [ 143.863351] Lustre: DEBUG MARKER: OSS: 2.17.58.2 [ 144.606500] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Sep 8 09:32:43 EDT 2026 [ 152.508880] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 153.150332] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 166.882323] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 166.883660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 166.888289] Lustre: Skipped 2 previous similar messages [ 166.891429] Lustre: Skipped 1 previous similar message [ 171.525664] Lustre: server umount lustre-MDT0000 complete [ 173.317216] LustreError: 6470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788874392 with bad export cookie 4655457133629403260 [ 173.318209] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 173.325152] LustreError: 6470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 177.119922] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 177.120155] LustreError: 6485:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 177.121161] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 177.121166] Lustre: Skipped 2 previous similar messages [ 177.125309] Lustre: Skipped 2 previous similar messages [ 177.135667] LustreError: 6485:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 178.145447] LustreError: 11483:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 178.146191] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 178.153242] LustreError: 11483:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 178.161019] Lustre: Skipped 2 previous similar messages [ 179.531979] Lustre: server umount lustre-MDT0001 complete [ 187.324348] Lustre: server umount lustre-OST0000 complete [ 203.231160] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 203.293403] Lustre: server umount lustre-OST0001 complete [ 205.761360] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_hostid [ 209.711461] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 224.560984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 224.685984] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 224.701583] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 224.742767] Lustre: lustre-MDT0000: new disk, initializing [ 224.772275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 224.780104] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 226.377181] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 231.584433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 231.638941] Lustre: 15028:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 231.768669] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 233.462961] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 237.870831] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 240.095417] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 242.770730] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 242.893367] Lustre: lustre-OST0000: new disk, initializing [ 242.895549] Lustre: Skipped 1 previous similar message [ 242.898497] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 242.902086] Lustre: Skipped 2 previous similar messages [ 242.905076] Lustre: 16997:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 242.948198] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 242.951164] Lustre: Skipped 1 previous similar message [ 244.358283] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 244.362141] Lustre: Skipped 1 previous similar message [ 244.364541] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 244.410350] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 245.200537] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 249.723091] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 251.439850] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 251.546821] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 253.330377] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 253.419933] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 254.947293] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 254.947778] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 254.952187] LustreError: Skipped 1 previous similar message [ 254.963568] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 254.971230] Lustre: Skipped 1 previous similar message [ 260.066434] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 260.070242] Lustre: Skipped 1 previous similar message [ 260.459339] Lustre: server umount lustre-OST0000 complete [ 262.624996] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 262.628423] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 262.632927] Lustre: Skipped 1 previous similar message [ 268.218162] Lustre: server umount lustre-MDT0000 complete [ 269.902633] LustreError: 15022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788874489 with bad export cookie 4655457133629404793 [ 269.905639] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 269.908327] LustreError: 15022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 270.082435] Lustre: server umount lustre-MDT0001 complete [ 271.593224] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 09:34:50 (1788874490) [ 274.368169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 274.576777] LustreError: 19259:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 274.610795] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 276.199786] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 279.942056] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 280.032218] LustreError: 19260:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 281.809474] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 284.628615] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 286.750722] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 289.480837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 292.327847] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 295.843678] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 305.121493] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 305.122528] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 305.135599] Lustre: Skipped 1 previous similar message [ 305.138791] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 305.141750] Lustre: Skipped 4 previous similar messages [ 306.484732] Lustre: server umount lustre-OST0000 complete [ 310.229786] Lustre: server umount lustre-MDT0000 complete [ 310.244490] LustreError: 19260:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 310.251845] LustreError: 19260:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 312.043044] LustreError: 19239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788874531 with bad export cookie 4655457133629405675 [ 312.045548] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 312.049909] LustreError: 19239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 312.215241] Lustre: server umount lustre-MDT0001 complete [ 317.046966] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 318.370357] Key type lgssc unregistered [ 318.543518] LNet: 22785:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 318.548294] LNetError: 22785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 318.557603] LNet: Removed LNI 192.168.206.131@tcp [ 318.999153] Key type .llcrypt unregistered [ 319.001299] Key type ._llcrypt unregistered [ 329.892454] Key type ._llcrypt registered [ 329.893788] Key type .llcrypt registered [ 329.934396] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 330.365084] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 330.425414] alg: No test for adler32 (adler32-zlib) [ 331.294364] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 331.406508] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 332.991169] Key type lgssc registered [ 333.474642] Lustre: Echo OBD driver; http://www.lustre.org/ [ 336.739922] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 336.752663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 337.954503] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 339.616143] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 343.104308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 343.269029] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 344.896753] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 347.887591] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 349.912888] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 352.589907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 352.736441] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 355.012574] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 357.745083] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 362.984791] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 373.219138] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 373.219373] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 373.220373] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 373.235483] Lustre: Skipped 1 previous similar message [ 375.488599] Lustre: server umount lustre-OST0000 complete [ 378.337478] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 378.338074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 378.344916] Lustre: Skipped 1 previous similar message [ 378.352101] Lustre: Skipped 2 previous similar messages [ 383.457737] LustreError: 24171:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 383.468733] LustreError: 24171:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 383.506723] Lustre: server umount lustre-MDT0000 complete [ 385.438422] LustreError: 26013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788874604 with bad export cookie 1249647064247574860 [ 385.439948] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 385.444691] LustreError: 26013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 385.619965] Lustre: server umount lustre-MDT0001 complete [ 391.405992] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 393.038583] Key type lgssc unregistered [ 393.227687] LNet: 28144:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 393.232714] LNetError: 28144:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 393.251550] LNet: Removed LNI 192.168.206.131@tcp [ 393.813567] Key type .llcrypt unregistered [ 393.816389] Key type ._llcrypt unregistered [ 406.660854] Key type ._llcrypt registered [ 406.662508] Key type .llcrypt registered [ 406.715860] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 407.067371] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 407.080399] alg: No test for adler32 (adler32-zlib) [ 407.982838] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 408.117326] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 409.735240] Key type lgssc registered [ 410.263768] Lustre: Echo OBD driver; http://www.lustre.org/ [ 413.843300] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 413.852812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 415.045590] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 416.778325] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 420.175840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 420.347948] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 422.087227] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 425.080191] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 427.468210] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 433.509666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 433.707569] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 436.404567] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 439.518701] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 440.871703] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 450.017491] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 450.018469] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 450.024907] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 450.032947] Lustre: Skipped 1 previous similar message [ 453.651698] Lustre: server umount lustre-OST0000 complete [ 455.137622] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 455.138589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 455.142929] Lustre: Skipped 2 previous similar messages [ 460.256504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 460.261289] Lustre: Skipped 2 previous similar messages [ 461.214015] Lustre: server umount lustre-MDT0000 complete [ 462.891826] LustreError: 29512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788874682 with bad export cookie 5418735057342365061 [ 462.892394] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 462.901492] LustreError: 29512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 463.047717] Lustre: server umount lustre-MDT0001 complete [ 467.619424] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 469.001092] Key type lgssc unregistered [ 469.183942] LNet: 33404:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 469.188405] LNetError: 33404:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 469.210242] LNet: Removed LNI 192.168.206.131@tcp [ 469.616343] Key type .llcrypt unregistered [ 469.618054] Key type ._llcrypt unregistered [ 480.942797] Key type ._llcrypt registered [ 480.944433] Key type .llcrypt registered [ 480.990406] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 481.398733] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 481.443169] alg: No test for adler32 (adler32-zlib) [ 482.332871] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 482.455411] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 484.055189] Key type lgssc registered [ 484.570652] Lustre: Echo OBD driver; http://www.lustre.org/ [ 488.308969] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 488.325419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 489.580127] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 491.351713] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 495.194475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 495.410442] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 497.142242] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 500.379568] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 502.783769] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 505.935093] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 506.127510] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 508.959158] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 512.424963] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 514.348747] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 529.887939] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 529.892922] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 529.901631] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 530.915154] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 530.915512] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 530.926671] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 530.928801] Lustre: Skipped 1 previous similar message [ 532.455507] Lustre: server umount lustre-OST0000 complete [ 536.034710] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 536.035924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 536.042176] Lustre: Skipped 1 previous similar message [ 536.046346] Lustre: Skipped 1 previous similar message [ 536.235232] Lustre: server umount lustre-MDT0000 complete [ 537.811507] LustreError: 37288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788874756 with bad export cookie 16897785391999891138 [ 537.813987] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 537.817144] LustreError: 37288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 544.008516] Lustre: server umount lustre-MDT0001 complete [ 548.568752] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 549.819316] Key type lgssc unregistered [ 549.968945] LNet: 38340:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 549.974429] LNetError: 38340:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 549.986108] LNet: Removed LNI 192.168.206.131@tcp [ 550.365387] Key type .llcrypt unregistered [ 550.367281] Key type ._llcrypt unregistered [ 561.281338] Key type ._llcrypt registered [ 561.282788] Key type .llcrypt registered [ 561.342157] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 561.801588] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 561.832697] alg: No test for adler32 (adler32-zlib) [ 562.705505] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 562.809570] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 564.399153] Key type lgssc registered [ 564.858713] Lustre: Echo OBD driver; http://www.lustre.org/ [ 568.225420] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 568.236515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 569.431797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 571.136200] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 574.694518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 574.890261] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 576.633363] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 579.896668] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 582.266358] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 585.079733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 585.240424] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 587.679847] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 590.501761] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 594.472381] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 604.643026] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 604.643064] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 604.644730] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 604.649311] Lustre: Skipped 1 previous similar message [ 604.652530] LustreError: Skipped 1 previous similar message [ 605.267480] Lustre: server umount lustre-OST0000 complete [ 620.001110] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 620.002289] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 620.007373] Lustre: Skipped 1 previous similar message [ 620.011935] Lustre: Skipped 2 previous similar messages [ 624.096593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 624.099531] Lustre: Skipped 1 previous similar message [ 629.216484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 629.219233] Lustre: Skipped 1 previous similar message [ 633.311131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 633.390059] Lustre: server umount lustre-MDT0000 complete [ 634.336460] LustreError: 39721:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 634.345573] LustreError: 39721:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 635.031470] LustreError: 39705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788874854 with bad export cookie 4343519248356262621 [ 635.032849] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 635.041265] LustreError: 39705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 641.248591] Lustre: server umount lustre-MDT0001 complete [ 646.121532] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 647.671528] Key type lgssc unregistered [ 647.863848] LNet: 43452:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 647.872188] LNetError: 43452:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 647.884114] LNet: Removed LNI 192.168.206.131@tcp [ 648.342344] Key type .llcrypt unregistered [ 648.344667] Key type ._llcrypt unregistered [ 659.566914] Key type ._llcrypt registered [ 659.568635] Key type .llcrypt registered [ 659.611112] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 659.966700] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 659.989982] alg: No test for adler32 (adler32-zlib) [ 660.858265] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 660.955863] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 662.551185] Key type lgssc registered [ 662.990118] Lustre: Echo OBD driver; http://www.lustre.org/ [ 666.404475] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 666.418072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 667.617187] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 669.391562] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 673.056510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 673.228430] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 674.933356] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 677.788800] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 679.997730] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 682.792703] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 682.942745] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 685.240379] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 688.626857] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 692.201492] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 697.313857] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 697.315038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 697.326026] Lustre: Skipped 2 previous similar messages [ 697.330470] Lustre: Skipped 1 previous similar message [ 699.461617] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.31@tcp (stopping) [ 699.466050] Lustre: Skipped 1 previous similar message [ 702.433328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 702.436975] Lustre: Skipped 2 previous similar messages [ 703.087695] Lustre: server umount lustre-MDT0000 complete [ 704.573458] LustreError: 47328:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 704.583780] LustreError: 47328:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 704.947685] LustreError: 44820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788874924 with bad export cookie 7517662628186878773 [ 704.951342] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 704.955903] LustreError: 44820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 707.558839] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 707.560391] LustreError: 44834:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 707.569587] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 709.599886] LustreError: 44835:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 714.720333] LustreError: 47328:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 714.720372] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 714.728901] LustreError: 47328:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 714.733758] Lustre: Skipped 2 previous similar messages [ 719.328943] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 719.422894] Lustre: server umount lustre-MDT0001 complete [ 731.625982] Lustre: server umount lustre-OST0000 complete [ 738.465508] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 739.787896] Key type lgssc unregistered [ 739.962828] LNet: 48696:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 739.967546] LNetError: 48696:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 739.976707] LNet: Removed LNI 192.168.206.131@tcp [ 740.395203] Key type .llcrypt unregistered [ 740.396777] Key type ._llcrypt unregistered [ 751.420477] Key type ._llcrypt registered [ 751.421973] Key type .llcrypt registered [ 751.470867] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 751.960587] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 751.972211] alg: No test for adler32 (adler32-zlib) [ 752.850763] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 752.970722] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 754.575161] Key type lgssc registered [ 755.082467] Lustre: Echo OBD driver; http://www.lustre.org/ [ 758.566265] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 758.576674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 774.047311] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 779.167298] LustreError: 50057:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.131@tcp: failed processing log, type 4: rc = -110 [ 805.791209] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 808.497369] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 811.687149] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 975.523856] Lustre: server umount lustre-OST0000 complete [ 983.130305] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 984.412095] Key type lgssc unregistered [ 984.567493] LNet: 51733:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 984.572580] LNetError: 51733:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 984.581421] LNet: Removed LNI 192.168.206.131@tcp [ 984.964694] Key type .llcrypt unregistered [ 984.966288] Key type ._llcrypt unregistered [ 996.626447] Key type ._llcrypt registered [ 996.628881] Key type .llcrypt registered [ 996.706546] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 997.156282] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 997.207223] alg: No test for adler32 (adler32-zlib) [ 998.117326] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 998.238608] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 999.839192] Key type lgssc registered [ 1000.319029] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1003.801810] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1003.812503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1005.110112] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1006.938920] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1011.024648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1011.282473] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1013.505700] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1017.250728] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1020.081587] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1020.295491] hrtimer: interrupt took 5433139 ns [ 1023.140846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1023.304230] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1025.878906] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1029.132343] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1030.139271] Lustre: server umount lustre-OST0000 complete [ 1035.937058] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1035.949095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1035.953726] Lustre: Skipped 1 previous similar message [ 1038.015952] Lustre: server umount lustre-MDT0000 complete [ 1039.655186] LustreError: 53100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788875258 with bad export cookie 8737759283927065417 [ 1039.657329] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1039.660708] LustreError: 53100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1039.818464] Lustre: server umount lustre-MDT0001 complete [ 1044.121621] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 1045.314742] Key type lgssc unregistered [ 1045.457695] LNet: 56648:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1045.461911] LNetError: 56648:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1045.472505] LNet: Removed LNI 192.168.206.131@tcp [ 1045.852804] Key type .llcrypt unregistered [ 1045.854214] Key type ._llcrypt unregistered [ 1056.555561] Key type ._llcrypt registered [ 1056.557125] Key type .llcrypt registered [ 1056.605684] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 1057.132512] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1057.147201] alg: No test for adler32 (adler32-zlib) [ 1058.035587] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 1058.161790] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 1059.759184] Key type lgssc registered [ 1060.368316] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1063.941942] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1063.951132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1079.455651] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1084.575503] LustreError: 58007:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.131@tcp: failed processing log, type 4: rc = -110 [ 1111.199180] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1113.497708] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1116.533193] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1119.291627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1119.500657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1119.525833] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1121.043815] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1124.474497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1124.664695] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1126.483877] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1129.607646] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1131.980757] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1135.008963] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1135.011902] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1135.014649] Lustre: Skipped 1 previous similar message [ 1140.195195] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1140.199963] Lustre: Skipped 2 previous similar messages [ 1145.314448] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1145.320841] Lustre: Skipped 1 previous similar message [ 1147.360318] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1147.428606] Lustre: server umount lustre-OST0000 complete [ 1163.758140] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1163.758674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1163.765177] Lustre: Skipped 1 previous similar message [ 1168.864950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1168.869158] Lustre: Skipped 2 previous similar messages [ 1178.079243] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1178.141664] Lustre: server umount lustre-MDT0000 complete [ 1179.704268] LustreError: 58013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788875398 with bad export cookie 711420300940319690 [ 1179.707814] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1179.709580] LustreError: 58013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1185.936802] Lustre: server umount lustre-MDT0001 complete [ 1190.777137] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 1192.198104] Key type lgssc unregistered [ 1192.353511] LNet: 61746:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1192.357444] LNetError: 61746:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1192.366389] LNet: Removed LNI 192.168.206.131@tcp [ 1192.763076] Key type .llcrypt unregistered [ 1192.765273] Key type ._llcrypt unregistered [ 1203.799792] Key type ._llcrypt registered [ 1203.801014] Key type .llcrypt registered [ 1203.847184] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 1204.205951] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1204.243400] alg: No test for adler32 (adler32-zlib) [ 1205.127952] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 1205.229865] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 1206.823137] Key type lgssc registered [ 1207.272128] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1210.319282] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1210.331881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1211.541947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1213.207799] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1216.808540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1217.024751] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1218.657817] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1221.720936] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1224.000260] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1226.655599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1226.829976] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1228.982973] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1231.724665] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1236.072277] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1246.690936] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1246.691128] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1246.698072] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1246.705535] Lustre: Skipped 1 previous similar message [ 1250.521785] Lustre: server umount lustre-OST0000 complete [ 1252.832247] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1252.836833] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1252.844649] Lustre: Skipped 1 previous similar message [ 1252.847375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1256.928618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1256.929057] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1258.168186] Lustre: server umount lustre-MDT0000 complete [ 1259.804564] LustreError: 63112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788875478 with bad export cookie 5196328610445274512 [ 1259.809392] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1259.811068] LustreError: 63112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1259.947415] Lustre: server umount lustre-MDT0001 complete [ 1264.319771] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 1265.517780] Key type lgssc unregistered [ 1265.656527] LNet: 66683:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1265.661033] LNetError: 66683:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1265.671690] LNet: Removed LNI 192.168.206.131@tcp [ 1266.029772] Key type .llcrypt unregistered [ 1266.031547] Key type ._llcrypt unregistered [ 1276.936420] Key type ._llcrypt registered [ 1276.937807] Key type .llcrypt registered [ 1276.982326] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 1277.374478] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1277.422759] alg: No test for adler32 (adler32-zlib) [ 1278.341498] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 1278.481523] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 1280.103215] Key type lgssc registered [ 1280.572218] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1283.824129] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 09:51:43 (1788875503) [ 1286.197952] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1286.204927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1287.368835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1288.771073] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1291.486874] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1291.616570] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1293.123897] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1295.838307] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1298.011943] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1300.632383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1300.771492] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1303.024991] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1305.654928] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1313.062409] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1318.758150] Lustre: Failing over lustre-MDT0000 [ 1318.961229] Lustre: server umount lustre-MDT0000 complete [ 1321.532362] LustreError: 68356:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1321.540427] LustreError: 68356:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1322.412842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1322.464317] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1322.465817] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1322.467616] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1322.477459] LustreError: 68361:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1322.533510] Lustre: *** cfs_fail_loc=135, val=0*** [ 1322.575263] LustreError: 71469:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 1322.578395] LustreError: 71469:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.131@tcp: cfg command failed: rc = -2 [ 1322.582445] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1322.582445] [ 1322.585977] LustreError: MGC192.168.206.131@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1322.592724] LustreError: 71457:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1322.596726] LustreError: 71457:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [ 1322.601956] LustreError: 71457:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 1322.605183] LustreError: 71457:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 1322.617156] Lustre: server umount lustre-MDT0000 complete [ 1322.619089] LustreError: 71457:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1323.489529] LustreError: 68362:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1323.490758] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1323.496115] LustreError: 68362:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1323.500672] Lustre: Skipped 1 previous similar message [ 1324.966261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1325.014704] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1325.109560] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1325.136142] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1326.581707] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1326.645896] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1330.148989] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1330.158160] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 1330.174401] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1332.704130] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1332.707386] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1332.711684] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1335.266985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1335.267263] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1335.268026] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1337.381585] Lustre: server umount lustre-OST0000 complete [ 1340.385374] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1340.385479] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1340.387128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1340.387134] Lustre: Skipped 1 previous similar message [ 1340.404650] Lustre: Skipped 1 previous similar message [ 1345.505242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1345.508718] Lustre: Skipped 2 previous similar messages [ 1350.624445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1350.627198] Lustre: Skipped 1 previous similar message [ 1353.183092] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1353.240222] Lustre: server umount lustre-MDT0000 complete [ 1354.476200] LustreError: 68343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788875573 with bad export cookie 2863240916332766900 [ 1354.479191] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1354.480614] LustreError: 68343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1354.584741] Lustre: server umount lustre-MDT0001 complete [ 1358.144769] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 1359.216152] Key type lgssc unregistered [ 1359.345386] LNet: 73245:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1359.348482] LNetError: 73245:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1359.358262] LNet: Removed LNI 192.168.206.131@tcp [ 1359.687372] Key type .llcrypt unregistered [ 1359.688858] Key type ._llcrypt unregistered [ 1368.943972] Key type ._llcrypt registered [ 1368.945260] Key type .llcrypt registered [ 1368.994874] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 1369.291093] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1369.352814] alg: No test for adler32 (adler32-zlib) [ 1370.212589] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 1370.321484] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 1371.911182] Key type lgssc registered [ 1372.336920] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1375.036584] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1375.045039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1376.206768] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1377.659911] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1380.947312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1381.111049] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1382.695985] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1385.392839] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1387.321396] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1389.742730] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1389.879570] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1391.895086] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1394.425802] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1401.126683] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 1407.270556] Lustre: Failing over lustre-MDT0000 [ 1407.479453] Lustre: server umount lustre-MDT0000 complete [ 1410.109131] LustreError: 75358:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1410.115950] LustreError: 75358:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1411.081292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1411.134785] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1411.202119] Lustre: *** cfs_fail_loc=a05, val=0*** [ 1411.265248] LustreError: 77734:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 1411.268263] LustreError: 77734:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.131@tcp: cfg command failed: rc = -9 [ 1411.272176] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1411.272176] [ 1411.276082] LustreError: MGC192.168.206.131@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1411.281925] LustreError: 77722:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 1411.285505] LustreError: 77722:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -9 [ 1411.289228] LustreError: 77722:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 1411.292029] LustreError: 77722:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 1411.304227] Lustre: server umount lustre-MDT0000 complete [ 1411.305905] LustreError: 77722:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 1411.553191] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1411.555895] LustreError: 74627:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1411.560281] Lustre: Skipped 1 previous similar message [ 1413.810847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1413.872673] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1413.986127] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1414.026122] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1415.221234] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1415.528227] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1419.241082] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1419.252126] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 1419.270354] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 1421.791947] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1421.795092] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1421.799936] Lustre: Skipped 1 previous similar message [ 1421.802291] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1424.354125] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1424.356919] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1424.357090] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1426.510726] Lustre: server umount lustre-OST0000 complete [ 1429.473048] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1429.473048] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1429.473864] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1429.476255] Lustre: Skipped 1 previous similar message [ 1429.480268] Lustre: Skipped 1 previous similar message [ 1434.592260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1434.595157] Lustre: Skipped 1 previous similar message [ 1439.712499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1439.714749] Lustre: Skipped 1 previous similar message [ 1442.271102] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1442.324957] Lustre: server umount lustre-MDT0000 complete [ 1443.536281] LustreError: 74613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788875662 with bad export cookie 13713244811635105078 [ 1443.539302] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1443.540525] LustreError: 74613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1443.650336] Lustre: server umount lustre-MDT0001 complete [ 1447.350213] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 1448.400368] Key type lgssc unregistered [ 1448.531507] LNet: 79512:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1448.535083] LNetError: 79512:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1448.544362] LNet: Removed LNI 192.168.206.131@tcp [ 1448.880129] Key type .llcrypt unregistered [ 1448.881422] Key type ._llcrypt unregistered [ 1452.967613] Key type ._llcrypt registered [ 1452.969486] Key type .llcrypt registered [ 1453.009371] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 1453.125444] Key type .llcrypt unregistered [ 1453.126730] Key type ._llcrypt unregistered [ 1462.094600] Key type ._llcrypt registered [ 1462.095989] Key type .llcrypt registered [ 1462.137649] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 1462.481789] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1462.509081] alg: No test for adler32 (adler32-zlib) [ 1463.380654] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 1463.482333] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 1465.071221] Key type lgssc registered [ 1465.496801] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1468.317350] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1468.326855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1469.483951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1470.944647] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1473.964031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1474.101656] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1475.553727] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1478.066726] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1479.901743] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1482.228838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1482.378822] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1484.509494] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1487.104950] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1495.715298] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 1501.154048] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1501.155923] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1501.156712] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1501.165267] Lustre: Skipped 1 previous similar message [ 1506.275351] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1506.277479] Lustre: Skipped 2 previous similar messages [ 1506.643955] Lustre: server umount lustre-OST0000 complete [ 1509.855974] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1509.858325] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1509.862177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1511.393526] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1514.074420] Lustre: server umount lustre-MDT0000 complete [ 1515.408239] LustreError: 81369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788875734 with bad export cookie 9141110556755347791 [ 1515.411586] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1515.412644] LustreError: 81369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1515.535614] Lustre: server umount lustre-MDT0001 complete [ 1519.293861] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 1520.460182] Key type lgssc unregistered [ 1520.608458] LNet: 84938:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1520.611802] LNetError: 84938:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1520.620383] LNet: Removed LNI 192.168.206.131@tcp [ 1520.971926] Key type .llcrypt unregistered [ 1520.973364] Key type ._llcrypt unregistered [ 1530.002599] Key type ._llcrypt registered [ 1530.004058] Key type .llcrypt registered [ 1530.049890] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 1530.553207] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1530.579242] alg: No test for adler32 (adler32-zlib) [ 1531.441204] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 1531.537154] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 1533.119134] Key type lgssc registered [ 1533.558035] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1536.439238] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1536.447710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1537.619679] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1539.152903] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1542.223631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1542.363766] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1543.845587] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1546.419386] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1548.335543] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1550.730936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1550.860939] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1552.929790] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1555.398504] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1562.147491] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 1572.322146] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1572.323830] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1572.325322] LustreError: Skipped 1 previous similar message [ 1572.331804] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1572.334906] Lustre: Skipped 1 previous similar message [ 1574.229346] Lustre: server umount lustre-OST0000 complete [ 1577.443234] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1577.444057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1577.447854] Lustre: Skipped 2 previous similar messages [ 1581.777510] Lustre: server umount lustre-MDT0000 complete [ 1582.560405] LustreError: 86323:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1582.565123] LustreError: 86323:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1583.186544] LustreError: 88823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788875802 with bad export cookie 3884437022953566573 [ 1583.188967] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1583.190622] LustreError: 88823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1583.313950] Lustre: server umount lustre-MDT0001 complete [ 1587.035926] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 1588.145700] Key type lgssc unregistered [ 1588.274387] LNet: 89877:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1588.277620] LNetError: 89877:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1588.285473] LNet: Removed LNI 192.168.206.131@tcp [ 1588.618979] Key type .llcrypt unregistered [ 1588.620745] Key type ._llcrypt unregistered [ 1597.584447] Key type ._llcrypt registered [ 1597.585836] Key type .llcrypt registered [ 1597.624549] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 1597.940758] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1597.971083] alg: No test for adler32 (adler32-zlib) [ 1598.821176] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 1598.915480] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 1600.495124] Key type lgssc registered [ 1600.884807] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1603.579331] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1603.587342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1604.774477] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1606.198723] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1609.073484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1609.220618] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1610.656830] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1613.236223] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1615.215780] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1617.502548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1617.635896] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1619.636873] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1622.100290] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1629.925465] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 1640.418169] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1640.418580] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1640.426447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1640.428509] Lustre: Skipped 1 previous similar message [ 1641.998338] Lustre: server umount lustre-OST0000 complete [ 1645.023668] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1645.025859] LustreError: Skipped 1 previous similar message [ 1645.026986] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1645.031232] Lustre: Skipped 1 previous similar message [ 1645.033349] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1649.364642] Lustre: server umount lustre-MDT0000 complete [ 1650.646221] LustreError: 91246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788875869 with bad export cookie 6903209504506932724 [ 1650.647587] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1650.651170] LustreError: 91246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1650.758781] Lustre: server umount lustre-MDT0001 complete [ 1654.489035] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 1655.552522] Key type lgssc unregistered [ 1655.684505] LNet: 94816:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1655.687247] LNetError: 94816:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1655.697500] LNet: Removed LNI 192.168.206.131@tcp [ 1656.008760] Key type .llcrypt unregistered [ 1656.010206] Key type ._llcrypt unregistered [ 1664.792586] Key type ._llcrypt registered [ 1664.793953] Key type .llcrypt registered [ 1664.837788] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 1665.318113] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1665.331894] alg: No test for adler32 (adler32-zlib) [ 1666.204365] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 1666.304651] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 1667.887121] Key type lgssc registered [ 1668.312128] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1671.096170] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1671.102898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1686.559387] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1691.679303] LustreError: 96175:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.131@tcp: failed processing log, type 4: rc = -110 [ 1718.303161] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1720.160693] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1722.419786] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1724.250729] Lustre: server umount lustre-OST0000 complete [ 1726.908189] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 09:59:06 (1788875946) [ 1728.995704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1729.130311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1730.434522] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1733.124662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1733.243679] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1734.560734] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1736.872568] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1738.599568] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1740.739628] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1740.860835] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1742.695236] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1745.127779] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1754.216079] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 1759.712365] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1759.713315] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1759.714488] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1759.723634] Lustre: Skipped 1 previous similar message [ 1761.165180] Lustre: server umount lustre-OST0000 complete [ 1764.319824] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1764.322536] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1764.327183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1764.329244] Lustre: Skipped 1 previous similar message [ 1764.565751] Lustre: server umount lustre-MDT0000 complete [ 1764.833792] LustreError: 97806:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1764.839225] LustreError: 97806:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1765.789100] LustreError: 97792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788875984 with bad export cookie 10209804015030991759 [ 1765.792366] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1765.793542] LustreError: 97792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1765.896820] Lustre: server umount lustre-MDT0001 complete [ 1769.385285] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 1770.386880] Key type lgssc unregistered [ 1770.507408] LNet: 101513:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1770.510364] LNetError: 101513:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1770.519363] LNet: Removed LNI 192.168.206.131@tcp [ 1770.833459] Key type .llcrypt unregistered [ 1770.834693] Key type ._llcrypt unregistered [ 1785.056119] Key type ._llcrypt registered [ 1785.057556] Key type .llcrypt registered [ 1785.095089] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 1785.445786] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1785.484025] alg: No test for adler32 (adler32-zlib) [ 1786.335719] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 1786.420185] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 1787.999173] Key type lgssc registered [ 1788.355324] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1790.817261] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1790.822213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1791.957634] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1793.316711] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1796.083879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1796.205877] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1797.549681] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1800.056125] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1801.934931] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1804.206169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1804.326906] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1806.244385] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1808.582282] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1817.641561] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 1822.688765] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1822.690246] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1822.690917] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1822.693703] Lustre: Skipped 1 previous similar message [ 1823.820846] Lustre: server umount lustre-OST0000 complete [ 1826.784236] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1826.787945] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1826.793140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1826.795046] Lustre: Skipped 1 previous similar message [ 1827.220561] Lustre: server umount lustre-MDT0000 complete [ 1827.808104] LustreError: 103670:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1827.808644] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1827.815178] LustreError: 103670:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1828.524447] LustreError: 103652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788876047 with bad export cookie 8804193365174842988 [ 1828.527988] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1828.529486] LustreError: 103652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1828.641889] Lustre: server umount lustre-MDT0001 complete [ 1832.457830] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 1833.563414] Key type lgssc unregistered [ 1833.707376] LNet: 107220:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1833.710213] LNetError: 107220:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1833.719420] LNet: Removed LNI 192.168.206.131@tcp [ 1834.053119] Key type .llcrypt unregistered [ 1834.054606] Key type ._llcrypt unregistered [ 1841.435808] Key type ._llcrypt registered [ 1841.437270] Key type .llcrypt registered [ 1841.476809] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 1841.835380] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1841.881659] alg: No test for adler32 (adler32-zlib) [ 1842.755322] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 1842.852050] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 1844.439141] Key type lgssc registered [ 1844.838108] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1847.529042] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1847.536383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1862.943331] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1868.063393] LustreError: 108449:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.131@tcp: failed processing log, type 4: rc = -110 [ 1894.687167] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1896.715095] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1899.074413] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1901.266620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1901.388150] LustreError: MGC192.168.206.131@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1901.393748] LustreError: 109471:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1901.397584] LustreError: 109471:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [ 1901.401296] LustreError: 109471:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 1901.431455] Lustre: server umount lustre-MDT0000 complete [ 1901.433613] LustreError: 109471:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1914.370392] Lustre: server umount lustre-OST0000 complete [ 1917.339996] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_hostid [ 1920.453500] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 1931.643971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1931.725703] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1931.738434] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1931.777614] Lustre: lustre-MDT0000: new disk, initializing [ 1931.799498] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1931.806057] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1933.185326] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1937.285657] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1937.324247] Lustre: 112257:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1937.348246] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1937.350610] Lustre: Skipped 1 previous similar message [ 1937.384663] Lustre: lustre-MDT0001: new disk, initializing [ 1937.416136] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1937.427964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1937.431922] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1938.725753] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1942.387080] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1944.235827] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1946.431184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1946.528110] Lustre: lustre-OST0000: new disk, initializing [ 1946.530788] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1946.533434] Lustre: 114228:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1946.563197] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1947.386544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1947.390540] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1947.427342] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1948.452784] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1951.981340] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1953.374556] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1953.444821] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1954.745253] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1954.819993] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1957.856461] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1957.857469] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1957.859389] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1957.862211] Lustre: Skipped 1 previous similar message [ 1961.546930] Lustre: server umount lustre-OST0000 complete [ 1962.976395] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1962.976566] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1962.977245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1962.977250] Lustre: Skipped 1 previous similar message [ 1962.979414] LustreError: Skipped 1 previous similar message [ 1962.989945] Lustre: Skipped 1 previous similar message [ 1968.096207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1968.097885] Lustre: Skipped 2 previous similar messages [ 1973.215964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1973.217795] Lustre: Skipped 1 previous similar message [ 1977.311107] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1977.366138] Lustre: server umount lustre-MDT0000 complete [ 1978.521989] LustreError: 113054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788876197 with bad export cookie 15167256473441084204 [ 1978.525022] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1978.526391] LustreError: 113054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1978.630960] Lustre: server umount lustre-MDT0001 complete [ 1981.200637] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 10:03:20 (1788876200) [ 1982.201881] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 1987.101488] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_hostid [ 1989.910517] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 2000.537409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2000.613192] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2000.622936] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2000.660237] Lustre: lustre-MDT0000: new disk, initializing [ 2000.682300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2000.688826] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2001.947890] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2005.901147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2005.940529] Lustre: 119122:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2005.960711] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2005.962309] Lustre: Skipped 1 previous similar message [ 2005.994917] Lustre: lustre-MDT0001: new disk, initializing [ 2006.035889] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2006.039375] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2007.264845] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2010.889905] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2012.593886] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2014.732742] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2014.821869] Lustre: lustre-OST0000: new disk, initializing [ 2014.823496] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2014.825634] Lustre: 121087:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2014.852028] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2014.854309] Lustre: Skipped 1 previous similar message [ 2015.924389] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2015.928644] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2015.955449] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2016.640594] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2020.241117] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2021.612179] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2021.680030] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2022.965170] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2023.031750] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2026.467163] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2026.467850] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2026.469314] LustreError: Skipped 1 previous similar message [ 2026.472970] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2026.476547] Lustre: Skipped 1 previous similar message [ 2029.779855] Lustre: server umount lustre-OST0000 complete [ 2031.584657] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2031.584832] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2031.591331] Lustre: Skipped 2 previous similar messages [ 2036.704034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2036.706353] Lustre: Skipped 3 previous similar messages [ 2037.075939] Lustre: server umount lustre-MDT0000 complete [ 2038.179947] LustreError: 119915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788876257 with bad export cookie 15167256473441085072 [ 2038.183448] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2038.183785] LustreError: 119915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2038.292945] Lustre: server umount lustre-MDT0001 complete [ 2040.935177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2041.063931] LustreError: 123297:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2041.069212] LustreError: 123297:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2041.084571] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2042.338305] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2045.015698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2046.317896] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2048.524320] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2050.191053] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2052.278904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2054.152238] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2056.413604] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2069.328025] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 2071.008329] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2071.010852] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2071.015560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2076.361952] Lustre: server umount lustre-OST0000 complete [ 2081.247773] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2081.250160] LustreError: Skipped 1 previous similar message [ 2081.251806] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2081.256944] Lustre: Skipped 1 previous similar message [ 2083.597542] Lustre: server umount lustre-MDT0000 complete [ 2084.723693] LustreError: 125906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788876303 with bad export cookie 15167256473441085940 [ 2084.726661] LustreError: 125906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2084.727963] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2084.842068] Lustre: server umount lustre-MDT0001 complete [ 2088.077470] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 2088.977548] Key type lgssc unregistered [ 2089.093328] LNet: 126945:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2089.097091] LNetError: 126945:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2089.105263] LNet: Removed LNI 192.168.206.131@tcp [ 2089.369081] Key type .llcrypt unregistered [ 2089.370488] Key type ._llcrypt unregistered [ 2094.783351] Key type ._llcrypt registered [ 2094.784978] Key type .llcrypt registered [ 2094.819568] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_hostid [ 2100.327544] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 2100.633541] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2100.673081] alg: No test for adler32 (adler32-zlib) [ 2101.519272] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 2101.605530] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 2103.183188] Key type lgssc registered [ 2103.489219] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2113.937980] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2113.944056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2115.038657] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2115.049085] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2115.085242] Lustre: lustre-MDT0000: new disk, initializing [ 2115.108865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2115.115456] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2116.511877] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2120.657977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2120.703181] Lustre: 130066:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2120.728966] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2120.731822] Lustre: Skipped 1 previous similar message [ 2120.769399] Lustre: lustre-MDT0001: new disk, initializing [ 2120.803272] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2120.818421] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2120.824644] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2122.192633] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2125.929240] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2127.782557] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2130.010242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2130.113601] Lustre: lustre-OST0000: new disk, initializing [ 2130.115294] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2130.118605] Lustre: 132034:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2130.148912] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2132.103746] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2135.795346] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2137.202382] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2139.638243] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2139.641168] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2139.672684] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2140.332860] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 2141.788806] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2141.866938] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2144.738250] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2144.738526] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2144.741452] LustreError: Skipped 1 previous similar message [ 2144.746342] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2144.749611] Lustre: Skipped 1 previous similar message [ 2148.628253] Lustre: server umount lustre-OST0000 complete [ 2151.391913] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2151.394883] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2151.400227] Lustre: Skipped 1 previous similar message [ 2151.403106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2154.976160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2154.976611] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2160.096154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2160.098346] Lustre: Skipped 2 previous similar messages [ 2164.191103] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2164.192192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2164.196988] Lustre: Skipped 1 previous similar message [ 2164.258953] Lustre: server umount lustre-MDT0000 complete [ 2165.540068] LustreError: 130862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788876384 with bad export cookie 7141299666017112230 [ 2165.542772] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2165.544248] LustreError: 130862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2165.664641] Lustre: server umount lustre-MDT0001 complete [ 2168.420488] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 10:06:27 (1788876387) [ 2170.521149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2170.670961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2171.883619] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2174.563366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2175.953062] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2178.195596] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2179.821664] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2184.544632] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2184.545325] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2184.548340] Lustre: Skipped 1 previous similar message [ 2186.579562] Lustre: server umount lustre-MDT0000 complete [ 2187.791028] LustreError: 134449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788876406 with bad export cookie 7141299666017113098 [ 2187.794448] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2187.796030] LustreError: 134449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2187.895106] Lustre: server umount lustre-MDT0001 complete [ 2190.466795] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 10:06:49 (1788876409) [ 2192.400617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2207.839590] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2212.959493] LustreError: 136881:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.131@tcp: failed processing log, type 4: rc = -110 [ 2238.559370] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2238.561334] Lustre: Skipped 1 previous similar message [ 2240.445940] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2242.924202] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2243.646633] Lustre: server umount lustre-OST0000 complete [ 2246.483545] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 10:07:45 (1788876465) [ 2248.628968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2248.797156] LustreError: 138326:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2248.804115] LustreError: 138326:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2248.822929] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2250.248894] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2252.910555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2254.238196] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2256.410234] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2258.079849] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2260.162417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2261.983386] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2264.137990] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2278.883155] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2278.885300] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2278.886026] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2278.886031] Lustre: Skipped 1 previous similar message [ 2278.886801] Lustre: Skipped 2 previous similar messages [ 2282.581242] Lustre: server umount lustre-MDT0000 complete [ 2283.838210] LustreError: 138354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788876502 with bad export cookie 7141299666017113749 [ 2283.841570] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2283.843204] LustreError: 138354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2283.965534] Lustre: server umount lustre-MDT0001 complete [ 2295.509834] Lustre: server umount lustre-OST0000 complete [ 2298.019682] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 10:08:37 (1788876517) [ 2299.965141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2300.100113] LustreError: 141996:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2300.121641] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2300.123620] Lustre: Skipped 2 previous similar messages [ 2301.383801] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2303.919240] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2305.355646] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2307.811545] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2309.633788] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2311.902042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2314.067139] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2316.579431] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2317.867583] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2325.349607] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 2326.059715] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec [ 2327.319425] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2327.389796] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2329.568240] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2329.570876] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2329.575166] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2329.577542] Lustre: Skipped 2 previous similar messages [ 2334.085910] Lustre: server umount lustre-OST0000 complete [ 2341.402697] Lustre: server umount lustre-MDT0000 complete [ 2342.565199] LustreError: 142021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788876561 with bad export cookie 7141299666017114806 [ 2342.569399] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2342.569409] LustreError: 142021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2342.690159] Lustre: server umount lustre-MDT0001 complete [ 2345.203968] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 10:09:24 (1788876564) [ 2347.259446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2362.723962] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2367.839291] LustreError: 146206:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.131@tcp: failed processing log, type 4: rc = -110 [ 2393.439361] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2393.441241] Lustre: Skipped 2 previous similar messages [ 2395.199914] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2397.437215] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2399.583621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2399.707117] LustreError: 146231:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2399.748092] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 2400.961651] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2403.566934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2404.887599] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2407.102769] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2408.874341] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2410.262855] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2410.340308] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2411.579749] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2411.648893] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2414.048456] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2414.050140] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2414.051890] Lustre: Skipped 3 previous similar messages [ 2414.054764] LustreError: Skipped 1 previous similar message [ 2414.056478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2414.059186] Lustre: Skipped 8 previous similar messages [ 2418.389238] Lustre: server umount lustre-MDT0000 complete [ 2419.168512] LustreError: 147242:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2419.173223] LustreError: 147242:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2419.502222] LustreError: 146212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788876638 with bad export cookie 7141299666017115814 [ 2419.504173] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2419.505736] LustreError: 146212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2419.601290] Lustre: server umount lustre-MDT0001 complete [ 2429.970177] Lustre: server umount lustre-OST0000 complete [ 2432.527081] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 10:10:51 (1788876651) [ 2434.548615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2449.951569] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2455.071515] LustreError: 150396:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.131@tcp: failed processing log, type 4: rc = -110 [ 2480.671316] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2480.673012] Lustre: Skipped 2 previous similar messages [ 2482.344453] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2484.729263] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2486.990632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2487.133182] LustreError: 150421:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2487.182598] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 2488.534946] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2491.444826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2492.905709] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2495.342249] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2497.352930] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2499.718209] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2499.788795] Lustre: lustre-OST0001: new disk, initializing [ 2499.790699] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2499.793331] Lustre: 153264:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2501.846239] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2503.801218] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2503.804702] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2503.848132] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2505.657853] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2507.171469] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2507.246109] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2508.641345] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2508.715077] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2512.352330] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2512.354661] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2512.358768] Lustre: Skipped 2 previous similar messages [ 2515.449516] Lustre: server umount lustre-OST0000 complete [ 2519.522025] LustreError: 150413:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2519.526643] LustreError: 150413:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2531.295108] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2531.348678] Lustre: server umount lustre-OST0001 complete [ 2532.831910] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2532.834406] LustreError: Skipped 3 previous similar messages [ 2538.709764] Lustre: server umount lustre-MDT0000 complete [ 2539.908142] LustreError: 155009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788876759 with bad export cookie 7141299666017116766 [ 2539.911693] LustreError: 155009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2539.911903] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2540.030664] Lustre: server umount lustre-MDT0001 complete [ 2547.606516] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 10:12:47 (1788876767) [ 2548.135857] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 2548.644727] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 2549.238270] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 10:12:48 (1788876768) [ 2551.474853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2551.571640] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2551.581581] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2553.063409] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2555.922614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2555.952286] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2555.963382] Lustre: 156823:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2557.533491] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2559.879259] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2561.672456] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2563.944900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2564.013158] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2566.002319] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2568.341592] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2569.739579] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2571.791745] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 2571.856963] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 2573.293165] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2573.366175] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2576.866988] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2576.867093] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2576.869735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2576.872781] Lustre: Skipped 6 previous similar messages [ 2576.875909] Lustre: Skipped 14 previous similar messages [ 2580.109963] Lustre: server umount lustre-OST0000 complete [ 2593.961437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2596.058538] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2597.158825] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2597.163920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2597.164044] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2597.171932] Lustre: Skipped 1 previous similar message [ 2598.559580] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2600.236398] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2600.305521] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2601.871178] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2601.962908] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2603.850156] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff893105cd7000.ost_server_uuid 50 [ 2604.366847] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff893105cd7000.ost_server_uuid in FULL state after 0 sec [ 2604.424992] LustreError: 156827:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.206.31@tcp) returned error from blocking AST (req@0000000011266036 x1875772776311808 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 000000004f4e708d/0x631afe35c390ebaa lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.206.31@tcp remote: 0xd97ab1352bf39cfa expref: 5 pid: 157942 timeout: 2704 lvb_type: 0 lru_score: 0 lru_type: 0 [ 2604.441369] LustreError: lustre-MDT0000: A client on nid 192.168.206.31@tcp was evicted due to a lock blocking callback time out: rc -107 [ 2604.445102] LustreError: 156818:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.206.31@tcp ns: mdt-lustre-MDT0000_UUID lock: 000000004f4e708d/0x631afe35c390ebaa lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.206.31@tcp remote: 0xd97ab1352bf39cfa expref: 6 pid: 157942 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 2611.598359] Lustre: server umount lustre-OST0000 complete [ 2619.100711] Lustre: server umount lustre-MDT0000 complete [ 2620.385552] LustreError: 156814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788876839 with bad export cookie 7141299666017117949 [ 2620.387857] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2620.389218] LustreError: 156814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2626.560757] Lustre: server umount lustre-MDT0001 complete [ 2630.359781] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 2631.408864] Key type lgssc unregistered [ 2631.534314] LNet: 162940:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2631.536658] LNetError: 162940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2631.543291] LNet: Removed LNI 192.168.206.131@tcp [ 2631.863742] Key type .llcrypt unregistered [ 2631.864916] Key type ._llcrypt unregistered [ 2640.774553] Key type ._llcrypt registered [ 2640.775786] Key type .llcrypt registered [ 2640.819542] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 2641.275744] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2641.300799] alg: No test for adler32 (adler32-zlib) [ 2642.173830] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 2642.276593] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 2643.855152] Key type lgssc registered [ 2644.229189] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2646.943356] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2646.950186] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2648.104472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2649.605624] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2652.509801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2652.638390] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2653.884178] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2656.213446] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2657.994741] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2660.125727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2660.236315] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2662.026956] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2664.285731] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2673.576407] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 2676.849436] Lustre: Failing over lustre-MDT0000 [ 2677.069992] Lustre: server umount lustre-MDT0000 complete [ 2678.239780] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2678.241943] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2678.245520] LustreError: 164326:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2678.250043] LustreError: 164326:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2678.752639] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2678.753791] LustreError: 166823:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2678.756745] Lustre: Skipped 1 previous similar message [ 2683.872287] LustreError: 164345:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2683.878210] LustreError: 164345:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2687.931218] LustreError: 164306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.206.31@tcp arrived at 1788876907 with bad export cookie 8135165176556601318 [ 2687.938222] LustreError: 164306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2688.993156] LustreError: 164327:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2688.998921] LustreError: 164327:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2690.348226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2690.393773] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2690.486918] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2690.511917] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2691.879581] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2692.614797] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2692.617059] Lustre: lustre-MDT0000: Denying connection for new client 64b16a3e-345a-4acb-b732-cc0ace1c03da (at 192.168.206.31@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2695.653965] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2700.770116] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2700.770818] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2700.773980] Lustre: Skipped 1 previous similar message [ 2705.549133] Lustre: server umount lustre-OST0000 complete [ 2706.771662] LustreError: 168081:0:(ldlm_lib.c:3029:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2706.774214] Lustre: 167433:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2706.776807] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2706.782746] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2706.786596] LustreError: 167433:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000aa059570 x1875773343435008/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2706.791720] LustreError: 167433:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 2706.794498] LustreError: 167433:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 2706.797409] Lustre: lustre-MDT0000: Recovery over after 0:14, of 2 clients 0 recovered and 2 were evicted. [ 2706.891350] Lustre: server umount lustre-MDT0000 complete [ 2708.209413] LustreError: 164308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788876927 with bad export cookie 8135165176556601675 [ 2708.214440] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2708.214504] LustreError: 163636:0:(import.c:751:ptlrpc_connect_import_locked()) can't connect to a closed import [ 2708.215422] LustreError: 164308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2714.420837] Lustre: server umount lustre-MDT0001 complete [ 2718.149207] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 2719.192338] Key type lgssc unregistered [ 2719.320338] LNet: 168831:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2719.322713] LNetError: 168831:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2719.330314] LNet: Removed LNI 192.168.206.131@tcp [ 2719.624708] Key type .llcrypt unregistered [ 2719.626308] Key type ._llcrypt unregistered [ 2728.641532] Key type ._llcrypt registered [ 2728.642627] Key type .llcrypt registered [ 2728.685829] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 2728.994328] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2729.008025] alg: No test for adler32 (adler32-zlib) [ 2729.870743] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 2729.967266] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 2731.551131] Key type lgssc registered [ 2731.937577] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2734.561159] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2734.568940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2735.720923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2737.192428] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2740.169198] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2740.317101] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2741.819218] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2744.278381] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2746.193555] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2748.501702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2748.635383] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2750.626884] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2753.164023] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2759.909369] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 2765.282207] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2765.282247] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2765.286191] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2765.289363] Lustre: Skipped 1 previous similar message [ 2766.984241] Lustre: server umount lustre-OST0000 complete [ 2770.194744] Lustre: server umount lustre-MDT0000 complete [ 2770.400354] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2770.400873] LustreError: 170213:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2770.403488] Lustre: Skipped 2 previous similar messages [ 2770.408974] LustreError: 170213:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2771.334268] LustreError: 170243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788876990 with bad export cookie 10273721995365551188 [ 2771.337363] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2771.339100] LustreError: 170243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2771.434701] Lustre: server umount lustre-MDT0001 complete [ 2774.811260] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 2775.703662] Key type lgssc unregistered [ 2775.826408] LNet: 173764:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2775.829282] LNetError: 173764:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2775.835418] LNet: Removed LNI 192.168.206.131@tcp [ 2776.109956] Key type .llcrypt unregistered [ 2776.111102] Key type ._llcrypt unregistered [ 2787.286737] Key type ._llcrypt registered [ 2787.287677] Key type .llcrypt registered [ 2787.321986] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 2787.719838] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2787.736666] alg: No test for adler32 (adler32-zlib) [ 2788.585343] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 2788.666770] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 2790.247191] Key type lgssc registered [ 2790.582405] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2793.030235] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2793.037624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2794.162743] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2795.377705] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2797.976021] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2798.087526] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2799.328512] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2801.480253] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2803.143342] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2805.314245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2805.415783] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2807.199439] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2809.572275] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2819.748122] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 2824.159942] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2824.162137] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2824.166890] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2825.187794] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2825.187909] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2825.188085] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2825.189701] Lustre: Skipped 1 previous similar message [ 2827.470195] Lustre: server umount lustre-OST0000 complete [ 2828.767730] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2828.770036] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2828.774455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2834.768414] Lustre: server umount lustre-MDT0000 complete [ 2835.423946] LustreError: 175437:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2835.430340] LustreError: 175437:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2835.938394] LustreError: 175463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788877055 with bad export cookie 16906839869357166331 [ 2835.941977] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2835.942811] LustreError: 175463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2836.040611] Lustre: server umount lustre-MDT0001 complete [ 2839.251669] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 2840.164954] Key type lgssc unregistered [ 2840.293378] LNet: 178989:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2840.295807] LNetError: 178989:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2840.303265] LNet: Removed LNI 192.168.206.131@tcp [ 2840.569410] Key type .llcrypt unregistered [ 2840.570376] Key type ._llcrypt unregistered [ 2848.592416] Key type ._llcrypt registered [ 2848.594011] Key type .llcrypt registered [ 2848.632371] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 2849.057500] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2849.081838] alg: No test for adler32 (adler32-zlib) [ 2849.926641] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 2850.015429] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 2851.599171] Key type lgssc registered [ 2851.964361] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2855.523755] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2855.529455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2856.651928] Lustre: *** cfs_fail_loc=135, val=0*** [ 2856.691369] LustreError: 180484:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2856.694423] LustreError: 180484:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.131@tcp: cfg command failed: rc = -2 [ 2856.697337] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2856.697337] [ 2856.700402] LustreError: MGC192.168.206.131@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2856.704726] LustreError: 180446:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2856.708038] LustreError: 180446:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [ 2856.710938] LustreError: 180446:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 2856.713582] LustreError: 180446:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2856.746530] Lustre: server umount lustre-MDT0000 complete [ 2856.747835] LustreError: 180446:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2858.943893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2859.104966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2860.467202] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2863.340973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2863.470506] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2864.852244] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2867.469384] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2869.356804] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2873.184942] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2873.189509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2873.191291] Lustre: Skipped 1 previous similar message [ 2876.246694] Lustre: server umount lustre-MDT0000 complete [ 2877.521675] LustreError: 180841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788877096 with bad export cookie 11676927858988363526 [ 2877.524855] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2877.525926] LustreError: 180841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2877.643229] Lustre: server umount lustre-MDT0001 complete [ 2881.341694] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 2882.417296] Key type lgssc unregistered [ 2882.553397] LNet: 183179:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2882.556186] LNetError: 183179:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2882.563306] LNet: Removed LNI 192.168.206.131@tcp [ 2882.858846] Key type .llcrypt unregistered [ 2882.860460] Key type ._llcrypt unregistered [ 2889.891174] Key type ._llcrypt registered [ 2889.892794] Key type .llcrypt registered [ 2889.936893] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 2890.051213] Key type .llcrypt unregistered [ 2890.052862] Key type ._llcrypt unregistered [ 2896.860136] Key type ._llcrypt registered [ 2896.861023] Key type .llcrypt registered [ 2896.900403] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 2897.280038] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2897.290569] alg: No test for adler32 (adler32-zlib) [ 2898.154060] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 2898.244652] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 2899.823166] Key type lgssc registered [ 2900.190088] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2902.839630] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2902.847312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2918.303383] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2923.423254] LustreError: 185280:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.131@tcp: failed processing log, type 4: rc = -110 [ 2950.047166] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2951.994301] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2954.260075] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2956.355913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2956.517028] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2956.540696] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2957.834809] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2960.447035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2960.562913] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2961.814182] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2963.974440] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2965.649600] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2966.517205] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 2972.127885] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2972.130062] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2972.134177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2975.201774] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2975.201874] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2977.286662] Lustre: server umount lustre-OST0000 complete [ 2980.320746] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2980.325824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2980.328645] Lustre: Skipped 2 previous similar messages [ 2984.538144] Lustre: server umount lustre-MDT0000 complete [ 2985.439895] LustreError: 186320:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2985.445432] LustreError: 186320:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2985.788224] LustreError: 185287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788877204 with bad export cookie 16768711338443533803 [ 2985.790990] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2985.793423] LustreError: 185287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2985.900321] Lustre: server umount lustre-MDT0001 complete [ 2989.676919] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 2990.718503] Key type lgssc unregistered [ 2990.847406] LNet: 189190:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2990.850224] LNetError: 189190:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2990.860280] LNet: Removed LNI 192.168.206.131@tcp [ 2991.176370] Key type .llcrypt unregistered [ 2991.177596] Key type ._llcrypt unregistered [ 3000.573178] Key type ._llcrypt registered [ 3000.574447] Key type .llcrypt registered [ 3000.612585] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 3001.013783] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3001.027135] alg: No test for adler32 (adler32-zlib) [ 3001.890844] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 3001.984420] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 3003.567132] Key type lgssc registered [ 3003.921714] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3006.493085] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3006.498961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3007.642553] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3009.099029] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 3012.046923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3012.184354] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3013.517732] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 3015.997115] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3017.953110] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3020.217507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3020.333481] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3022.278437] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 3024.687562] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3033.638040] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 3036.391195] Lustre: Failing over lustre-MDT0000 [ 3036.491857] Lustre: server umount lustre-MDT0000 complete [ 3037.664230] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3037.667516] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3037.672403] LustreError: 191996:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3037.677017] LustreError: 191996:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3038.688514] LustreError: 190575:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3038.689883] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3038.695739] LustreError: 190575:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3038.699365] Lustre: Skipped 1 previous similar message [ 3040.315255] LustreError: 192416:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3043.808375] LustreError: 190576:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3043.814440] LustreError: 190576:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3048.928990] LustreError: 191996:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3048.933436] LustreError: 191996:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3049.325332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3049.367636] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3049.471595] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3049.500103] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3050.550525] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3050.941513] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 3052.548436] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 3054.565929] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3054.574511] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 3054.590540] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 3055.762934] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 3060.961375] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3060.964580] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3060.969776] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3063.264139] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3063.267643] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3063.271934] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3064.803101] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3064.805332] Lustre: Skipped 1 previous similar message [ 3067.032428] Lustre: server umount lustre-OST0000 complete [ 3069.920481] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3069.920482] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3069.921103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3069.929236] Lustre: Skipped 1 previous similar message [ 3074.452996] Lustre: server umount lustre-MDT0000 complete [ 3075.041589] LustreError: 190575:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3075.048258] LustreError: 190575:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3075.648843] LustreError: 190600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788877294 with bad export cookie 4340171552835953402 [ 3075.650657] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3075.652989] LustreError: 190600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3075.765473] Lustre: server umount lustre-MDT0001 complete [ 3079.070631] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 3080.057724] Key type lgssc unregistered [ 3080.177335] LNet: 195426:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3080.179398] LNetError: 195426:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3080.186235] LNet: Removed LNI 192.168.206.131@tcp [ 3080.460067] Key type .llcrypt unregistered [ 3080.461101] Key type ._llcrypt unregistered [ 3088.654155] Key type ._llcrypt registered [ 3088.655221] Key type .llcrypt registered [ 3088.689560] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 3088.992341] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3088.999453] alg: No test for adler32 (adler32-zlib) [ 3089.837232] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 3089.913395] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 3091.487105] Key type lgssc registered [ 3091.787564] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3094.156180] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3094.163801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3095.336219] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3096.836376] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 3099.525859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3099.635921] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3100.891517] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 3103.095928] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3104.758070] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3106.825063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3106.939543] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3108.853300] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 3111.351330] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3120.230359] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 3122.340499] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 3127.836908] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 3138.579324] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3150.303884] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3150.306148] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3150.312233] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3151.329146] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3151.330589] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3151.331111] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3151.340369] Lustre: Skipped 1 previous similar message [ 3153.870762] Lustre: server umount lustre-OST0000 complete [ 3155.935924] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3155.939556] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3155.944731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3161.307543] Lustre: server umount lustre-MDT0000 complete [ 3161.567902] LustreError: 196811:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3161.573225] LustreError: 196811:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3162.555168] LustreError: 196792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788877381 with bad export cookie 1373018257966389730 [ 3162.557600] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3162.561271] LustreError: 196792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3162.695693] Lustre: server umount lustre-MDT0001 complete [ 3166.493742] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 3167.622445] Key type lgssc unregistered [ 3167.749493] LNet: 200678:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3167.751929] LNetError: 200678:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3167.759452] LNet: Removed LNI 192.168.206.131@tcp [ 3168.130059] Key type .llcrypt unregistered [ 3168.131190] Key type ._llcrypt unregistered [ 3175.908308] Key type ._llcrypt registered [ 3175.909364] Key type .llcrypt registered [ 3175.953697] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 3176.444195] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3176.454125] alg: No test for adler32 (adler32-zlib) [ 3177.315393] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 3177.418935] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 3179.007181] Key type lgssc registered [ 3179.407984] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3182.210568] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3182.216793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3183.381801] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3184.827941] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 3187.762470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3187.914856] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3189.351541] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 3191.762490] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3193.537608] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3195.704022] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3195.826634] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3197.665790] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 3200.049274] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3208.106555] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 3212.272208] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 3221.158624] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 3230.341000] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 3235.410233] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 3245.023949] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3245.026563] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3245.031451] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3249.122271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3249.122904] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3249.123167] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3250.650622] Lustre: server umount lustre-OST0000 complete [ 3254.240599] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3254.241315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3254.244049] Lustre: Skipped 1 previous similar message [ 3254.245726] Lustre: Skipped 1 previous similar message [ 3258.004979] Lustre: server umount lustre-MDT0000 complete [ 3259.206172] LustreError: 204616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788877478 with bad export cookie 15727751018984733209 [ 3259.209896] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3259.210405] LustreError: 204616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3259.326665] Lustre: server umount lustre-MDT0001 complete [ 3262.852018] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 3263.802517] Key type lgssc unregistered [ 3263.928304] LNet: 207386:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3263.930614] LNetError: 207386:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3263.948305] LNet: Removed LNI 192.168.206.131@tcp [ 3264.219420] Key type .llcrypt unregistered [ 3264.220334] Key type ._llcrypt unregistered [ 3272.520593] Key type ._llcrypt registered [ 3272.521521] Key type .llcrypt registered [ 3272.555480] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 3272.830701] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3272.894646] alg: No test for adler32 (adler32-zlib) [ 3273.740073] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 3273.825992] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 3275.407131] Key type lgssc registered [ 3275.745976] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3278.132802] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3278.138731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3279.272424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3280.554546] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 3283.209441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3283.330781] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3284.553885] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 3286.733805] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3288.363261] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3290.478359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3290.617537] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3292.387499] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 3294.578078] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3303.910523] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 3309.025493] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3309.027706] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3309.029941] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3309.031439] LustreError: Skipped 1 previous similar message [ 3309.035751] Lustre: Skipped 1 previous similar message [ 3310.157787] Lustre: server umount lustre-OST0000 complete [ 3313.573072] Lustre: server umount lustre-MDT0000 complete [ 3314.145663] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3314.145915] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3314.146545] LustreError: 208767:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3314.146552] LustreError: 208767:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3314.160166] Lustre: Skipped 2 previous similar messages [ 3314.863492] LustreError: 211274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788877534 with bad export cookie 7212621893230194279 [ 3314.867131] LustreError: 211274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3314.868987] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3314.998463] Lustre: server umount lustre-MDT0001 complete [ 3318.802969] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 3319.914312] Key type lgssc unregistered [ 3320.052553] LNet: 212470:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3320.056164] LNetError: 212470:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3320.063420] LNet: Removed LNI 192.168.206.131@tcp [ 3320.407996] Key type .llcrypt unregistered [ 3320.409114] Key type ._llcrypt unregistered [ 3327.945243] Key type ._llcrypt registered [ 3327.946976] Key type .llcrypt registered [ 3327.990686] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 3328.378563] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3328.395764] alg: No test for adler32 (adler32-zlib) [ 3329.267086] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 3329.369789] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 3330.951166] Key type lgssc registered [ 3331.319625] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3333.974816] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3333.981655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3335.143188] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3336.548686] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 3339.635206] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3339.770951] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3341.160896] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 3343.601596] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3345.563944] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3347.982758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3348.116295] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3350.094913] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 3352.496538] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3360.424505] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 3369.951894] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3369.954424] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3369.959389] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3370.977130] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3370.977454] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3370.978506] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3376.098559] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3376.101395] Lustre: Skipped 2 previous similar messages [ 3380.191196] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3380.250326] Lustre: server umount lustre-OST0000 complete [ 3385.824133] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3385.826678] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3385.831699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3387.679594] Lustre: server umount lustre-MDT0000 complete [ 3388.942230] LustreError: 214443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788877608 with bad export cookie 18114517763564924691 [ 3388.946058] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3388.947360] LustreError: 214443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3389.074413] Lustre: server umount lustre-MDT0001 complete [ 3392.837226] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 3393.961673] Key type lgssc unregistered [ 3394.093437] LNet: 217410:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3394.095781] LNetError: 217410:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3394.107362] LNet: Removed LNI 192.168.206.131@tcp [ 3394.428963] Key type .llcrypt unregistered [ 3394.430212] Key type ._llcrypt unregistered [ 3401.860125] Key type ._llcrypt registered [ 3401.861367] Key type .llcrypt registered [ 3401.902278] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 3402.285370] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3402.345155] alg: No test for adler32 (adler32-zlib) [ 3403.216191] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 3403.311712] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 3404.895155] Key type lgssc registered [ 3405.276582] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3407.775240] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3407.780302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3408.927095] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3410.397949] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 3413.281731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3413.409695] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3414.803131] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 3417.199233] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3419.117865] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3421.276569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3421.404917] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3423.393328] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 3425.828958] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3434.726047] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 3439.111724] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3439.146734] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3439.243768] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3441.128715] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 3443.417502] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3445.903689] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec [ 3456.255974] Lustre: Permanently deactivating lustre-OST0001 [ 3456.261073] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3465.202721] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 3468.249111] Lustre: Permanently reactivating lustre-OST0001 [ 3468.254823] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 3468.257348] Lustre: Skipped 2 previous similar messages [ 3475.449626] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3475.454036] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3475.456929] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3475.461219] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3475.477170] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3480.544657] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3480.544771] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3480.548224] LustreError: Skipped 1 previous similar message [ 3480.552806] Lustre: Skipped 1 previous similar message [ 3480.555962] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3480.559327] Lustre: Skipped 1 previous similar message [ 3481.572786] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3481.575207] Lustre: Skipped 1 previous similar message [ 3484.670410] Lustre: server umount lustre-OST0001 complete [ 3486.175795] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3486.178360] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3486.182965] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3486.689087] LustreError: 220470:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3486.693619] LustreError: 220470:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3491.810549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3491.812766] LustreError: 221141:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3491.813451] Lustre: Skipped 2 previous similar messages [ 3492.051923] Lustre: server umount lustre-OST0000 complete [ 3495.391785] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3495.394055] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3495.398079] Lustre: Skipped 1 previous similar message [ 3496.932773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3496.935994] Lustre: Skipped 1 previous similar message [ 3499.480805] Lustre: server umount lustre-MDT0000 complete [ 3500.625209] LustreError: 218632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788877719 with bad export cookie 15618158519278125837 [ 3500.627672] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3500.629805] LustreError: 218632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3500.745903] Lustre: server umount lustre-MDT0001 complete [ 3504.323610] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 3505.368078] Key type lgssc unregistered [ 3505.508345] LNet: 223715:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3505.510599] LNetError: 223715:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3505.517428] LNet: Removed LNI 192.168.206.131@tcp [ 3505.817795] Key type .llcrypt unregistered [ 3505.819277] Key type ._llcrypt unregistered [ 3518.888779] Key type ._llcrypt registered [ 3518.889631] Key type .llcrypt registered [ 3518.923385] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 3519.226473] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3519.258576] alg: No test for adler32 (adler32-zlib) [ 3520.116791] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 3520.201135] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 3521.783177] Key type lgssc registered [ 3522.127778] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3524.617904] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3524.624389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3525.709024] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3525.715313] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3525.769780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3526.998337] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 3529.600977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3529.635177] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3529.645932] Lustre: 225891:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3529.736528] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3531.012098] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 3533.308547] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3535.160559] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3537.435599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3537.524605] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3537.601198] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3538.109501] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 3539.637745] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 3541.949380] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3543.516057] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 3551.076881] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 3558.609040] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 3569.444743] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3585.607285] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3585.610863] Lustre: Skipped 1 previous similar message [ 3611.524683] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 3611.527512] Lustre: Skipped 2 previous similar messages [ 3650.252712] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3650.255062] Lustre: Skipped 4 previous similar messages [ 3717.601734] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3717.603190] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3717.603734] LustreError: Skipped 1 previous similar message [ 3717.607352] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3717.609879] Lustre: Skipped 1 previous similar message [ 3719.750552] Lustre: server umount lustre-OST0000 complete [ 3722.721388] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3722.722300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3722.724874] Lustre: Skipped 2 previous similar messages [ 3722.727930] Lustre: Skipped 1 previous similar message [ 3727.130776] Lustre: server umount lustre-MDT0000 complete [ 3728.402507] LustreError: 225882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788877947 with bad export cookie 18268604342533867283 [ 3728.406037] LustreError: 225882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3728.407222] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3728.542583] Lustre: server umount lustre-MDT0001 complete [ 3732.272425] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 3733.375121] Key type lgssc unregistered [ 3733.517364] LNet: 230739:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3733.519828] LNetError: 230739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3733.529334] LNet: Removed LNI 192.168.206.131@tcp [ 3733.866430] Key type .llcrypt unregistered [ 3733.867716] Key type ._llcrypt unregistered [ 3742.547049] Key type ._llcrypt registered [ 3742.547989] Key type .llcrypt registered [ 3742.591519] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 3742.914943] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3742.942579] alg: No test for adler32 (adler32-zlib) [ 3743.795315] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 3743.881813] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 3745.463144] Key type lgssc registered [ 3745.775620] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3748.185400] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3748.191680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3749.337258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3750.577389] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 3753.313885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3753.439379] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3754.722593] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 3756.987688] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3758.763090] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3761.002157] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3761.118558] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3763.082218] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 3765.540118] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3774.437808] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 3788.767863] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3788.769918] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3788.773618] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3789.792519] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3789.792703] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3789.793063] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3789.800824] Lustre: Skipped 1 previous similar message [ 3792.015316] Lustre: server umount lustre-OST0000 complete [ 3794.399733] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3794.402032] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3794.405614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3799.323787] Lustre: server umount lustre-MDT0000 complete [ 3800.031835] LustreError: 232137:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3800.037486] LustreError: 232137:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3800.479113] LustreError: 234738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788878019 with bad export cookie 14253269043815384034 [ 3800.482292] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3800.483477] LustreError: 234738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3800.593936] Lustre: server umount lustre-MDT0001 complete [ 3804.023609] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 3804.936110] Key type lgssc unregistered [ 3805.058439] LNet: 235844:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3805.060295] LNetError: 235844:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3805.067370] LNet: Removed LNI 192.168.206.131@tcp [ 3805.361550] Key type .llcrypt unregistered [ 3805.362974] Key type ._llcrypt unregistered [ 3813.356098] Key type ._llcrypt registered [ 3813.357018] Key type .llcrypt registered [ 3813.392261] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 3813.678967] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3813.716789] alg: No test for adler32 (adler32-zlib) [ 3814.554442] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 3814.635302] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 3816.215116] Key type lgssc registered [ 3816.571548] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4083.416410] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 4084.475062] Key type lgssc unregistered [ 4084.606291] LNet: 237842:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4084.609230] LNetError: 237842:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4084.615366] LNet: Removed LNI 192.168.206.131@tcp [ 4084.905311] Key type .llcrypt unregistered [ 4084.906944] Key type ._llcrypt unregistered [ 4092.486399] Key type ._llcrypt registered [ 4092.487616] Key type .llcrypt registered [ 4092.522606] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_hostid [ 4098.669625] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 4099.034570] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4099.054820] alg: No test for adler32 (adler32-zlib) [ 4099.895502] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 4099.979093] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 4101.551086] Key type lgssc registered [ 4101.903413] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4111.316556] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 3966 sec ========== 10:38:50 (1788878330) [ 4111.849827] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 10:38:51 (1788878331) === [ 4113.005391] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 10:38:52 (1788878332) === [ 4125.179917] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 4126.161599] Key type lgssc unregistered [ 4126.286317] LNet: 243044:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4126.289563] LNetError: 243044:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4126.301236] LNet: Removed LNI 192.168.206.131@tcp [ 4126.571356] Key type .llcrypt unregistered [ 4126.572350] Key type ._llcrypt unregistered