[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 456248511 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.003088] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005011] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.010128] LSM: Security Framework initializing [ 0.011072] Yama: becoming mindful. [ 0.012026] SELinux: Initializing. [ 0.013060] *** VALIDATE selinux *** [ 0.020640] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024532] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025119] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026113] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027091] *** VALIDATE tmpfs *** [ 0.028316] *** VALIDATE proc *** [ 0.029182] *** VALIDATE cgroup *** [ 0.030007] *** VALIDATE cgroup2 *** [ 0.031194] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.032111] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.033006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.034028] Spectre V2 : User space: Vulnerable [ 0.036004] Speculative Store Bypass: Vulnerable [ 0.039330] debug: unmapping init [mem 0xffffffffa4c59000-0xffffffffa4c60fff] [ 0.041152] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042727] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043022] ... version: 2 [ 0.044010] ... bit width: 48 [ 0.045011] ... generic registers: 4 [ 0.046012] ... value mask: 0000ffffffffffff [ 0.047010] ... max period: 00007fffffffffff [ 0.048010] ... fixed-purpose events: 3 [ 0.049010] ... event mask: 000000070000000f [ 0.050349] rcu: Hierarchical SRCU implementation. [ 0.052583] smp: Bringing up secondary CPUs ... [ 0.053608] x86: Booting SMP configuration: [ 0.054027] .... node #0, CPUs: #1 #2 #3 [ 0.057326] smp: Brought up 1 node, 4 CPUs [ 0.059012] smpboot: Max logical packages: 1 [ 0.060013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.127879] node 0 deferred pages initialised in 65ms [ 0.130114] devtmpfs: initialized [ 0.131259] x86/mm: Memory block size: 128MB [ 0.134760] gcov: version magic: 0x41383552 [ 0.137394] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.141095] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.144410] pinctrl core: initialized pinctrl subsystem [ 0.146182] [ 0.146958] ************************************************************* [ 0.150016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.153011] ** ** [ 0.155009] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.157008] ** ** [ 0.160012] ** This means that this kernel is built to expose internal ** [ 0.162009] ** IOMMU data structures, which may compromise security on ** [ 0.164015] ** your system. ** [ 0.167013] ** ** [ 0.170029] ** If you see this message and you are not debugging the ** [ 0.172010] ** kernel, report this immediately to your vendor! ** [ 0.175049] ** ** [ 0.177013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.180011] ************************************************************* [ 0.183726] NET: Registered protocol family 16 [ 0.185506] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.188050] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.190051] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.194014] cpuidle: using governor menu [ 0.196010] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.198730] PCI: Using configuration type 1 for base access [ 0.201124] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.211245] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.212000] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.213255] cryptd: max_cpu_qlen set to 1000 [ 0.214237] ACPI: Added _OSI(Module Device) [ 0.216012] ACPI: Added _OSI(Processor Device) [ 0.218011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.220012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.225852] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.234663] ACPI: Interpreter enabled [ 0.236037] ACPI: PM: (supports S0 S3 S4 S5) [ 0.237012] ACPI: Using IOAPIC for interrupt routing [ 0.240145] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.244428] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.255934] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.257034] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.259095] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.264130] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.270380] acpiphp: Slot [2] registered [ 0.272137] acpiphp: Slot [5] registered [ 0.273122] acpiphp: Slot [6] registered [ 0.275221] acpiphp: Slot [7] registered [ 0.277190] acpiphp: Slot [8] registered [ 0.279168] acpiphp: Slot [9] registered [ 0.281265] acpiphp: Slot [10] registered [ 0.284155] acpiphp: Slot [3] registered [ 0.285142] acpiphp: Slot [4] registered [ 0.287088] acpiphp: Slot [11] registered [ 0.289080] acpiphp: Slot [12] registered [ 0.291118] acpiphp: Slot [13] registered [ 0.293103] acpiphp: Slot [14] registered [ 0.295113] acpiphp: Slot [15] registered [ 0.296295] acpiphp: Slot [16] registered [ 0.298092] acpiphp: Slot [17] registered [ 0.300123] acpiphp: Slot [18] registered [ 0.302143] acpiphp: Slot [19] registered [ 0.304085] acpiphp: Slot [20] registered [ 0.306098] acpiphp: Slot [21] registered [ 0.307070] acpiphp: Slot [22] registered [ 0.308080] acpiphp: Slot [23] registered [ 0.310087] acpiphp: Slot [24] registered [ 0.312115] acpiphp: Slot [25] registered [ 0.314168] acpiphp: Slot [26] registered [ 0.316109] acpiphp: Slot [27] registered [ 0.317072] acpiphp: Slot [28] registered [ 0.318070] acpiphp: Slot [29] registered [ 0.320081] acpiphp: Slot [30] registered [ 0.322117] acpiphp: Slot [31] registered [ 0.324059] PCI host bridge to bus 0000:00 [ 0.325018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.329037] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.333025] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.336022] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.339022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.343055] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.345158] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.349311] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.352506] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.365014] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.370021] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.372015] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.374020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.376016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.379395] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.382794] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.384027] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.387984] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.393013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.406016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.411017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.417346] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.424016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.430015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.451013] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.457000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.464012] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.468013] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.480018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.489834] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.495014] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.501014] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.514016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.522000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.527013] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.531011] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.541013] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.548748] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.553012] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.557019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.569017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.577511] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.582014] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.587012] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.602011] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.609793] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.612359] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.614481] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.616287] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.618133] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.621202] iommu: Default domain type: Passthrough [ 0.623651] SCSI subsystem initialized [ 0.626123] ACPI: bus type USB registered [ 0.627069] usbcore: registered new interface driver usbfs [ 0.629054] usbcore: registered new interface driver hub [ 0.630053] usbcore: registered new device driver usb [ 0.631238] pps_core: LinuxPPS API ver. 1 registered [ 0.632007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.634045] PTP clock support registered [ 0.636110] EDAC MC: Ver: 3.0.0 [ 0.637241] PCI: Using ACPI for IRQ routing [ 0.638849] NetLabel: Initializing [ 0.640008] NetLabel: domain hash size = 128 [ 0.641005] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.642068] NetLabel: unlabeled traffic allowed by default [ 0.645081] vgaarb: loaded [ 0.646407] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.648015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.654233] clocksource: Switched to clocksource kvm-clock [ 0.754771] VFS: Disk quotas dquot_6.6.0 [ 0.756451] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.758410] *** VALIDATE ramfs *** [ 0.759710] *** VALIDATE hugetlbfs *** [ 0.761140] pnp: PnP ACPI init [ 0.763637] pnp: PnP ACPI: found 6 devices [ 0.778816] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.781272] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.783713] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.786356] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.788773] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.790697] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.793127] NET: Registered protocol family 2 [ 0.795378] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.799941] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.802757] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.806757] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.809201] TCP: Hash tables configured (established 65536 bind 65536) [ 0.811303] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.813467] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.815310] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.817269] NET: Registered protocol family 1 [ 0.821401] RPC: Registered named UNIX socket transport module. [ 0.823071] RPC: Registered udp transport module. [ 0.824312] RPC: Registered tcp transport module. [ 0.825394] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.827132] NET: Registered protocol family 44 [ 0.828213] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.831075] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.832514] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.833956] PCI: CLS 0 bytes, default 64 [ 0.835197] Unpacking initramfs... [ 2.367570] debug: unmapping init [mem 0xffff91ad3cc54000-0xffff91ad3ffbffff] [ 2.375338] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.377610] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.381160] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.199927] Initialise system trusted keyrings [ 3.202042] Key type blacklist registered [ 3.204425] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.216884] zbud: loaded [ 3.220972] *** VALIDATE nfs *** [ 3.222765] *** VALIDATE nfs4 *** [ 3.225471] pstore: using deflate compression [ 3.231522] Platform Keyring initialized [ 3.448101] NET: Registered protocol family 38 [ 3.449747] Key type asymmetric registered [ 3.451820] Asymmetric key parser 'x509' registered [ 3.453739] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.457370] io scheduler mq-deadline registered [ 3.459247] io scheduler kyber registered [ 3.461213] io scheduler bfq registered [ 3.463487] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.466842] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.470829] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.473925] ACPI: Power Button [PWRF] [ 3.480107] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.488113] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.501211] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.508784] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.541901] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.570514] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.600093] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.605816] Non-volatile memory driver v1.3 [ 3.608048] Linux agpgart interface v0.103 [ 3.645212] virtio_blk virtio1: [vda] 134160 512-byte logical blocks (68.7 MB/65.5 MiB) [ 3.648753] vda: detected capacity change from 0 to 68689920 [ 3.687368] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.690360] vdb: detected capacity change from 0 to 1073741824 [ 3.708333] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.711282] vdc: detected capacity change from 0 to 2621440000 [ 3.729337] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.732638] vdd: detected capacity change from 0 to 2621440000 [ 3.750315] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.753820] vde: detected capacity change from 0 to 4294967296 [ 3.775634] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.779401] vdf: detected capacity change from 0 to 4294967296 [ 3.787752] libphy: Fixed MDIO Bus: probed [ 3.799078] usbcore: registered new interface driver usbserial_generic [ 3.821503] usbserial: USB Serial support registered for generic [ 3.827593] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.834803] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.838405] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.842300] mousedev: PS/2 mouse device common for all mice [ 3.850546] rtc_cmos 00:05: RTC can wake from S4 [ 3.855088] rtc_cmos 00:05: registered as rtc0 [ 3.857473] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.861905] intel_pstate: CPU model not supported [ 3.865879] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.868528] hid: raw HID events driver (C) Jiri Kosina [ 3.883563] usbcore: registered new interface driver usbhid [ 3.887015] usbhid: USB HID core driver [ 3.889875] drop_monitor: Initializing network drop monitor service [ 3.893606] Initializing XFRM netlink socket [ 3.897305] NET: Registered protocol family 10 [ 3.899902] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.905971] Segment Routing with IPv6 [ 3.913045] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.913526] NET: Registered protocol family 17 [ 3.920396] mpls_gso: MPLS GSO support [ 3.927811] RAS: Correctable Errors collector initialized. [ 3.931305] AVX version of gcm_enc/dec engaged. [ 3.933670] AES CTR mode by8 optimization enabled [ 4.140917] sched_clock: Marking stable (4140848282, 0)->(4962687549, -821839267) [ 4.147701] registered taskstats version 1 [ 4.151363] Loading compiled-in X.509 certificates [ 4.169907] zswap: loaded using pool lzo/zbud [ 4.250632] Key type big_key registered [ 4.264206] Key type encrypted registered [ 4.265865] ima: No TPM chip found, activating TPM-bypass! [ 4.268214] ima: Allocated hash algorithm: sha1 [ 4.270139] ima: No architecture policies found [ 4.271928] evm: Initialising EVM extended attributes: [ 4.274553] evm: security.selinux [ 4.276229] evm: security.ima [ 4.277497] evm: security.capability [ 4.278840] evm: HMAC attrs: 0x1 [ 4.282036] rtc_cmos 00:05: setting system clock to 2026-04-11 05:07:18 UTC (1775884038) [ 4.289905] debug: unmapping init [mem 0xffffffffa5c03000-0xffffffffa5dfffff] [ 4.293203] debug: unmapping init [mem 0xffffffffa4982000-0xffffffffa4c58fff] [ 4.298056] Write protecting the kernel read-only data: 28672k [ 4.304475] debug: unmapping init [mem 0xffffffffa3003000-0xffffffffa31fffff] [ 4.308854] debug: unmapping init [mem 0xffffffffa3914000-0xffffffffa39fffff] [ 4.415242] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.425808] systemd[1]: Detected virtualization kvm. [ 4.428402] systemd[1]: Detected architecture x86-64. [ 4.431119] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.461243] systemd[1]: No hostname configured. [ 4.463224] systemd[1]: Set hostname to . [ 4.465763] random: systemd: uninitialized urandom read (16 bytes read) [ 4.468581] systemd[1]: Initializing machine ID from random generator. [ 5.392762] random: systemd: uninitialized urandom read (16 bytes read) [ 5.394630] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 5.398367] random: systemd: uninitialized urandom read (16 bytes read) [ 5.400615] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 5.484614] random: systemd: uninitialized urandom read (16 bytes read) [ 5.488495] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ 5.507634] random: systemd: uninitialized urandom read (16 bytes read) [ OK ] Reached target Paths. [ 5.524338] random: systemd: uninitialized urandom read (16 bytes read) [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... Starting Journal Service... [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.821926] device-mapper: uevent: version 1.0.3 [ 7.824733] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 11.632613] virtio_net virtio0 ens2: renamed from eth0 [ 11.776653] random: fast init done [ 13.532812] scsi host0: ata_piix [ 13.591915] scsi host1: ata_piix [ 13.640305] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 13.667678] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 18.802173] random: crng init done [ 18.803872] random: 4 urandom warning(s) missed due to ratelimiting [ 21.129215] dracut-initqueue[583]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 23.088030] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 25.741178] printk: systemd: 26 output lines suppressed due to ratelimiting [ 26.574783] SELinux: Disabled at runtime. [ 26.706497] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 26.739547] systemd[1]: Detected virtualization kvm. [ 26.743975] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 28.630309] systemd[1]: initrd-switch-root.service: Succeeded. [ 28.649052] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 28.660428] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 28.670445] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 28.681662] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 28.729246] systemd[1]: Starting Journal Service... Starting Journal Service... [ 28.763607] systemd[1]: proc-sys-fs-binfmt_misc.automount: Refusing to start, unit to trigger not loaded. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on RPCbind Server Activation Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Remount Root and Kernel File Systems... Mounting Kernel Debug File System... [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting POSIX Message Queue File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Starting Apply Kernel Variables... Mounting Huge Pages File System... [ OK ] Created slice system-getty.slice. Starting udev Coldplug all Devices... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on Process Core Dump Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Journal Service. [ 29.613631] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 30.699392] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 31.924166] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 31.957443] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 32.636947] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 32.753402] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit)[ 37.697365] Key type dns_resolver registered [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 38.279255] NFS: Registering the id_resolver key type [ 38.281260] Key type id_resolver registered [ 38.282826] Key type id_legacy registered [ *** ] A start job is running for Configur…only root support (10s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ 45.996634] hrtimer: interrupt took 11627588 ns [ OK ] Started Hostname Service. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Started Authorization Manager. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg231-server login: [ 86.448202] libcfs: loading out-of-tree module taints kernel. [ 86.473275] Key type ._llcrypt registered [ 86.474695] Key type .llcrypt registered [ 86.547817] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_hostid [ 98.300455] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 99.242793] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 99.257771] alg: No test for adler32 (adler32-zlib) [ 100.296930] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 100.694143] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 102.327175] Key type lgssc registered [ 103.580512] Lustre: Echo OBD driver; http://www.lustre.org/ [ 114.136052] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 140.737103] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 149.020753] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 149.055511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 150.267728] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 150.307444] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 150.362163] Lustre: lustre-MDT0000: new disk, initializing [ 150.428076] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 150.444092] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 152.871667] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 162.671319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 162.733992] Lustre: 6476:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 162.758709] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 162.763352] Lustre: Skipped 1 previous similar message [ 162.814164] Lustre: lustre-MDT0001: new disk, initializing [ 162.861686] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 162.884673] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 162.894549] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 165.239061] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 168.661510] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 174.540181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 174.667509] Lustre: lustre-OST0000: new disk, initializing [ 174.671601] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 174.726179] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 176.769460] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 176.774706] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 176.808505] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 177.597318] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 186.578206] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 186.637949] Lustre: lustre-OST0001: new disk, initializing [ 186.643356] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 186.692511] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 191.026419] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 192.536601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 192.541783] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 192.589457] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 200.076661] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 204.232561] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 213.363180] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing check_logdir /tmp/testlogs/ [ 217.079430] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing yml_node [ 220.088404] Lustre: DEBUG MARKER: Client: 2.17.52.3 [ 222.156168] Lustre: DEBUG MARKER: MDS: 2.17.52.3 [ 223.932850] Lustre: DEBUG MARKER: OSS: 2.17.52.3 [ 225.035407] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Apr 11 01:10:58 EDT 2026 [ 237.016585] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 237.986891] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 253.920738] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 253.922303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 253.926034] Lustre: Skipped 2 previous similar messages [ 253.935576] Lustre: Skipped 3 previous similar messages [ 257.184317] Lustre: server umount lustre-MDT0000 complete [ 259.746877] LustreError: 6467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775884293 with bad export cookie 6934541353534211994 [ 259.749322] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 259.751947] LustreError: 6467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 264.162935] LustreError: 6484:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 264.163266] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 264.164361] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 264.172098] LustreError: 6484:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 264.197914] Lustre: Skipped 2 previous similar messages [ 266.025200] Lustre: server umount lustre-MDT0001 complete [ 268.392589] Lustre: server umount lustre-OST0000 complete [ 271.193477] Lustre: server umount lustre-OST0001 complete [ 275.850126] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_hostid [ 281.898320] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 306.549908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 306.704434] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 306.736972] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 306.806250] Lustre: lustre-MDT0000: new disk, initializing [ 306.872642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 306.890364] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 309.561823] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 315.789987] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 315.855376] Lustre: 14943:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 315.922350] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 315.929407] Lustre: Skipped 1 previous similar message [ 315.990375] Lustre: lustre-MDT0001: new disk, initializing [ 316.046240] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 316.090396] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 316.102767] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 318.534331] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 323.598902] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 325.944746] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 330.302233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 331.851841] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 331.954864] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 334.206136] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 339.261950] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 342.031122] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 342.141685] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 344.340491] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 344.472084] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 347.616391] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 347.617182] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 347.618966] LustreError: Skipped 1 previous similar message [ 347.625000] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 347.627270] Lustre: Skipped 2 previous similar messages [ 351.694478] Lustre: server umount lustre-OST0000 complete [ 357.344214] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 357.349541] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 357.357303] Lustre: Skipped 1 previous similar message [ 357.366290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 362.979884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 362.982351] Lustre: Skipped 2 previous similar messages [ 368.095306] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 368.170254] Lustre: server umount lustre-MDT0000 complete [ 370.161639] LustreError: 14934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775884404 with bad export cookie 6934541353534213534 [ 370.166456] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 370.170307] LustreError: 14934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 370.362272] Lustre: server umount lustre-MDT0001 complete [ 372.637652] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 01:13:26 (1775884406) [ 376.362834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 376.629524] LustreError: 18800:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 376.681473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 376.686071] Lustre: Skipped 1 previous similar message [ 379.404283] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 381.921162] LustreError: 18801:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 384.670573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 387.431952] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 390.923969] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 393.015970] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 397.166897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 401.407314] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 404.942828] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 421.366685] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 421.368647] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 421.381174] Lustre: Skipped 1 previous similar message [ 421.384114] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 421.394787] Lustre: Skipped 3 previous similar messages [ 425.197257] Lustre: server umount lustre-OST0000 complete [ 431.074637] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 431.082118] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 431.087189] Lustre: Skipped 1 previous similar message [ 440.801859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 440.807233] Lustre: Skipped 5 previous similar messages [ 442.335316] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 442.490025] Lustre: server umount lustre-MDT0000 complete [ 444.931252] LustreError: 18781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775884479 with bad export cookie 6934541353534214416 [ 444.932478] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 444.942772] LustreError: 18781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 445.166689] Lustre: server umount lustre-MDT0001 complete [ 451.876911] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 453.640933] Key type lgssc unregistered [ 453.870051] LNet: 21996:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 453.875775] LNetError: 21996:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 454.951828] LNet: Removed LNI 192.168.202.131@tcp [ 455.540276] Key type .llcrypt unregistered [ 455.547211] Key type ._llcrypt unregistered [ 470.587120] Key type ._llcrypt registered [ 470.589078] Key type .llcrypt registered [ 470.680706] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 471.407288] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 471.418301] alg: No test for adler32 (adler32-zlib) [ 472.392463] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 472.545560] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 474.135199] Key type lgssc registered [ 474.786225] Lustre: Echo OBD driver; http://www.lustre.org/ [ 479.832917] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 479.872234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 481.323993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 484.274223] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 490.575812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 490.835331] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 494.144991] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 498.239186] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 500.984961] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 505.792844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 506.070348] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 507.126513] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 510.235268] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 514.080706] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 522.722253] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 522.726206] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 522.735234] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 525.795551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 525.795402] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 525.820853] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 528.889682] Lustre: server umount lustre-OST0000 complete [ 531.942707] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 531.948416] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 531.960468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 531.966804] Lustre: Skipped 1 previous similar message [ 536.035683] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 536.035719] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 539.773761] Lustre: server umount lustre-MDT0000 complete [ 541.152744] LustreError: 23370:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 541.166905] LustreError: 23370:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 542.142610] LustreError: 26257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775884576 with bad export cookie 7252692374511966099 [ 542.150764] LustreError: 26257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 542.151717] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 542.434916] Lustre: server umount lustre-MDT0001 complete [ 548.924402] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 550.612589] Key type lgssc unregistered [ 550.851713] LNet: 26990:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 550.856346] LNetError: 26990:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 550.871270] LNet: Removed LNI 192.168.202.131@tcp [ 551.488950] Key type .llcrypt unregistered [ 551.490324] Key type ._llcrypt unregistered [ 565.146886] Key type ._llcrypt registered [ 565.149067] Key type .llcrypt registered [ 565.231622] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 565.791336] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 565.802070] alg: No test for adler32 (adler32-zlib) [ 566.744166] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 566.937306] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 568.560227] Key type lgssc registered [ 569.238490] Lustre: Echo OBD driver; http://www.lustre.org/ [ 573.628315] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 573.650960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 574.972289] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 577.051714] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 581.784800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 581.995817] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 584.225451] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 587.626237] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 589.745988] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 597.011407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 597.287574] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 600.365044] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 600.664711] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 603.807118] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 615.904696] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 615.908916] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 615.915414] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 617.952595] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 617.956967] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 617.959804] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 619.689786] Lustre: server umount lustre-OST0000 complete [ 623.073364] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 623.077268] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 623.078848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 623.078853] Lustre: Skipped 1 previous similar message [ 623.080144] Lustre: Skipped 1 previous similar message [ 628.195607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 628.201612] Lustre: Skipped 2 previous similar messages [ 633.312718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 633.317341] Lustre: Skipped 1 previous similar message [ 636.383299] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 636.486920] Lustre: server umount lustre-MDT0000 complete [ 638.434133] LustreError: 29073:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 638.442810] LustreError: 29073:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 638.567824] LustreError: 28344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775884672 with bad export cookie 16955171167722952396 [ 638.574559] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 638.582621] LustreError: 28344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 638.737913] Lustre: server umount lustre-MDT0001 complete [ 644.696201] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 646.480645] Key type lgssc unregistered [ 646.658504] LNet: 31883:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 646.661372] LNetError: 31883:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 646.674972] LNet: Removed LNI 192.168.202.131@tcp [ 647.174803] Key type .llcrypt unregistered [ 647.176294] Key type ._llcrypt unregistered [ 661.502362] Key type ._llcrypt registered [ 661.504328] Key type .llcrypt registered [ 661.558266] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 662.112898] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 662.170728] alg: No test for adler32 (adler32-zlib) [ 663.135379] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 663.278578] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 664.879248] Key type lgssc registered [ 665.643638] Lustre: Echo OBD driver; http://www.lustre.org/ [ 670.100489] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 670.116540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 671.451298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 674.016473] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 679.964058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 680.232317] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 682.849555] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 686.440447] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 688.675688] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 692.899642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 693.154375] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 696.242138] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 696.544427] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 700.585957] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 710.626565] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 710.628734] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 710.629893] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 710.638574] Lustre: Skipped 1 previous similar message [ 712.331263] Lustre: server umount lustre-OST0000 complete [ 715.747619] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 715.752807] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 715.758554] Lustre: Skipped 1 previous similar message [ 715.769090] Lustre: Skipped 2 previous similar messages [ 720.857775] Lustre: server umount lustre-MDT0000 complete [ 720.867873] LustreError: 33253:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 720.888307] LustreError: 33253:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 723.039783] LustreError: 33974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775884757 with bad export cookie 15993182798160123323 [ 723.042554] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 723.045735] LustreError: 33974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 723.290726] Lustre: server umount lustre-MDT0001 complete [ 730.117686] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 731.773485] Key type lgssc unregistered [ 731.944922] LNet: 36437:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 731.951425] LNetError: 36437:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 731.966960] LNet: Removed LNI 192.168.202.131@tcp [ 732.525606] Key type .llcrypt unregistered [ 732.526990] Key type ._llcrypt unregistered [ 746.975252] Key type ._llcrypt registered [ 746.976812] Key type .llcrypt registered [ 747.063666] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 747.536797] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 747.573320] alg: No test for adler32 (adler32-zlib) [ 748.503677] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 748.602373] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 750.240203] Key type lgssc registered [ 750.897449] Lustre: Echo OBD driver; http://www.lustre.org/ [ 755.161679] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 755.177592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 756.461828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 758.517882] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 763.471800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 763.692315] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 766.004717] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 769.049715] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 771.223351] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 775.686431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 775.903839] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 779.116620] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 781.040771] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 782.421844] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 791.520741] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 791.525158] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 791.532852] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 793.568877] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 793.572940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 793.575267] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 793.584069] Lustre: Skipped 1 previous similar message [ 796.328426] Lustre: server umount lustre-OST0000 complete [ 811.556715] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 811.562544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 811.573183] Lustre: Skipped 1 previous similar message [ 811.583296] Lustre: Skipped 2 previous similar messages [ 813.601687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 813.605740] Lustre: Skipped 1 previous similar message [ 817.792599] Lustre: server umount lustre-MDT0000 complete [ 818.661223] LustreError: 38522:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 818.678386] LustreError: 38522:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 820.114723] LustreError: 37792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775884854 with bad export cookie 11694023876570284580 [ 820.120186] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 820.123241] LustreError: 37792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 826.435724] Lustre: server umount lustre-MDT0001 complete [ 832.262713] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 834.009322] Key type lgssc unregistered [ 834.245964] LNet: 41160:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 834.250088] LNetError: 41160:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 835.301746] LNet: Removed LNI 192.168.202.131@tcp [ 835.859551] Key type .llcrypt unregistered [ 835.861604] Key type ._llcrypt unregistered [ 849.304374] Key type ._llcrypt registered [ 849.305883] Key type .llcrypt registered [ 849.375670] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 849.859499] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 849.872163] alg: No test for adler32 (adler32-zlib) [ 850.830652] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 850.981143] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 852.615663] Key type lgssc registered [ 853.372879] Lustre: Echo OBD driver; http://www.lustre.org/ [ 858.205932] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 858.219903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 859.521631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 862.229724] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 867.546219] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 867.729610] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 870.000173] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 873.682220] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 876.230822] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 880.799429] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 881.052116] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 884.589978] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 885.175792] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 888.141644] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 897.513745] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 897.515559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 897.526576] Lustre: Skipped 2 previous similar messages [ 897.528486] Lustre: Skipped 1 previous similar message [ 900.728931] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.31@tcp (stopping) [ 900.733742] Lustre: Skipped 1 previous similar message [ 902.627786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 902.631140] Lustre: Skipped 2 previous similar messages [ 903.155231] Lustre: server umount lustre-MDT0000 complete [ 905.361891] LustreError: 42514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775884939 with bad export cookie 17147511995547882503 [ 905.363708] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 905.368731] LustreError: 42514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 905.837561] LustreError: 42528:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 905.845917] LustreError: 42528:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 905.855673] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.31@tcp (stopping) [ 907.751740] LustreError: 42530:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 907.758787] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 910.755615] LustreError: 42529:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 910.756691] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 910.774388] Lustre: Skipped 1 previous similar message [ 911.716275] Lustre: server umount lustre-MDT0001 complete [ 930.352807] Lustre: server umount lustre-OST0000 complete [ 937.879603] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 939.185366] Key type lgssc unregistered [ 939.341901] LNet: 46039:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 939.346974] LNetError: 46039:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 939.356515] LNet: Removed LNI 192.168.202.131@tcp [ 939.782833] Key type .llcrypt unregistered [ 939.784785] Key type ._llcrypt unregistered [ 953.239834] Key type ._llcrypt registered [ 953.241881] Key type .llcrypt registered [ 953.282482] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 953.733294] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 953.772101] alg: No test for adler32 (adler32-zlib) [ 954.699865] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 954.798047] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 956.391285] Key type lgssc registered [ 956.939606] Lustre: Echo OBD driver; http://www.lustre.org/ [ 960.809831] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 960.822317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 976.287424] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 981.407266] LustreError: 47388:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.131@tcp: failed processing log, type 4: rc = -110 [ 1008.031163] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1010.581576] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1012.809724] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1176.807962] Lustre: server umount lustre-OST0000 complete [ 1183.197151] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 1184.358271] Key type lgssc unregistered [ 1184.500989] LNet: 48931:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1184.503967] LNetError: 48931:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1184.512295] LNet: Removed LNI 192.168.202.131@tcp [ 1184.830121] Key type .llcrypt unregistered [ 1184.831316] Key type ._llcrypt unregistered [ 1194.858990] Key type ._llcrypt registered [ 1194.860750] Key type .llcrypt registered [ 1194.902828] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 1195.342675] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1195.351523] alg: No test for adler32 (adler32-zlib) [ 1196.223922] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 1196.326583] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 1197.927169] Key type lgssc registered [ 1198.367845] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1201.472293] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1201.483338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1202.671645] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1204.256231] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1208.330761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1208.514407] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1210.111111] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1212.184127] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1213.593940] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1216.766087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1216.900830] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1218.998826] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1221.130925] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1223.137353] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1223.138579] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1223.147966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1227.234095] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1228.241684] Lustre: server umount lustre-OST0000 complete [ 1232.354494] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1232.356084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1232.359862] Lustre: Skipped 1 previous similar message [ 1232.365487] Lustre: Skipped 2 previous similar messages [ 1237.472559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1237.474939] Lustre: Skipped 1 previous similar message [ 1242.592300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1242.595636] Lustre: Skipped 1 previous similar message [ 1244.127153] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1244.200025] Lustre: server umount lustre-MDT0000 complete [ 1245.782787] LustreError: 50282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775885280 with bad export cookie 6424947549607777132 [ 1245.784345] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1245.788744] LustreError: 50282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1245.921140] Lustre: server umount lustre-MDT0001 complete [ 1250.394503] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 1251.641032] Key type lgssc unregistered [ 1251.788940] LNet: 53476:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1251.793143] LNetError: 53476:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1251.799342] LNet: Removed LNI 192.168.202.131@tcp [ 1252.143140] Key type .llcrypt unregistered [ 1252.144605] Key type ._llcrypt unregistered [ 1262.740203] Key type ._llcrypt registered [ 1262.741733] Key type .llcrypt registered [ 1262.787783] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 1263.245051] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1263.251737] alg: No test for adler32 (adler32-zlib) [ 1264.144918] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 1264.251390] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 1265.847178] Key type lgssc registered [ 1266.274292] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1269.365952] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1269.376746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1284.831417] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1289.951208] LustreError: 54824:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.131@tcp: failed processing log, type 4: rc = -110 [ 1316.575214] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1318.603160] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1320.714960] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1323.240759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1323.426781] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1323.446952] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1324.881652] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1328.077226] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1328.253269] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1329.792976] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1331.952900] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1333.509634] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1336.802402] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1336.803308] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1336.807534] Lustre: Skipped 1 previous similar message [ 1340.453376] Lustre: server umount lustre-OST0000 complete [ 1356.311501] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1356.311967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1356.316087] Lustre: Skipped 1 previous similar message [ 1356.318499] Lustre: Skipped 1 previous similar message [ 1357.344545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1357.347240] Lustre: Skipped 2 previous similar messages [ 1362.400658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1362.403872] Lustre: Skipped 1 previous similar message [ 1362.562867] Lustre: server umount lustre-MDT0000 complete [ 1364.068118] LustreError: 57228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775885398 with bad export cookie 1575647481275466651 [ 1364.070881] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1364.072464] LustreError: 57228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1370.238132] Lustre: server umount lustre-MDT0001 complete [ 1374.287574] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 1375.367621] Key type lgssc unregistered [ 1375.502350] LNet: 58216:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1375.506109] LNetError: 58216:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1375.513456] LNet: Removed LNI 192.168.202.131@tcp [ 1375.827520] Key type .llcrypt unregistered [ 1375.828899] Key type ._llcrypt unregistered [ 1386.170485] Key type ._llcrypt registered [ 1386.171831] Key type .llcrypt registered [ 1386.214503] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 1386.553794] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1386.608062] alg: No test for adler32 (adler32-zlib) [ 1387.480446] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 1387.579220] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 1389.167163] Key type lgssc registered [ 1389.599282] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1392.670428] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1392.683558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1393.897435] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1395.447270] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1399.027605] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1399.187867] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1400.852065] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1403.130494] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1404.798932] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1407.545442] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1407.736707] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1410.339787] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1412.873117] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1418.984471] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1429.474048] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1429.474806] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1429.481508] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1429.485958] Lustre: Skipped 1 previous similar message [ 1432.909922] Lustre: server umount lustre-OST0000 complete [ 1434.592293] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1434.593494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1434.596872] Lustre: Skipped 2 previous similar messages [ 1439.712053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1439.716923] Lustre: Skipped 2 previous similar messages [ 1440.609269] Lustre: server umount lustre-MDT0000 complete [ 1441.950945] LustreError: 61758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775885476 with bad export cookie 2135336562637394542 [ 1441.954817] LustreError: 61758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1441.954882] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1442.077332] Lustre: server umount lustre-MDT0001 complete [ 1446.276224] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 1447.378629] Key type lgssc unregistered [ 1447.530065] LNet: 62793:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1447.533471] LNetError: 62793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1447.542507] LNet: Removed LNI 192.168.202.131@tcp [ 1447.885894] Key type .llcrypt unregistered [ 1447.889839] Key type ._llcrypt unregistered [ 1459.856234] Key type ._llcrypt registered [ 1459.857862] Key type .llcrypt registered [ 1459.904590] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 1460.266723] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1460.325541] alg: No test for adler32 (adler32-zlib) [ 1461.194552] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 1461.296867] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 1462.887790] Key type lgssc registered [ 1463.295247] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1466.762804] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 01:31:40 (1775885500) [ 1469.529437] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1469.538578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1470.714942] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1472.220241] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1475.536741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1475.702535] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1477.302416] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1479.397207] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1480.852133] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1483.424194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1483.632840] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1485.969688] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1488.284172] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1495.911280] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1503.029470] Lustre: Failing over lustre-MDT0000 [ 1503.180759] Lustre: server umount lustre-MDT0000 complete [ 1505.912040] LustreError: 64452:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1505.918936] LustreError: 64452:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1506.274463] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1506.279799] Lustre: Skipped 1 previous similar message [ 1506.876669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1506.931499] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1506.985873] Lustre: *** cfs_fail_loc=135, val=0*** [ 1507.025835] LustreError: 67228:0:(obd_config.c:843:class_setup()) setup lustre-MDT0000 failed (-2) [ 1507.028260] LustreError: 67228:0:(obd_config.c:2101:class_config_llog_handler()) MGC192.168.202.131@tcp: cfg command failed: rc = -2 [ 1507.031593] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1507.031593] [ 1507.036255] LustreError: MGC192.168.202.131@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1507.044035] LustreError: 67217:0:(tgt_mount.c:1819:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1507.048095] LustreError: 67217:0:(tgt_mount.c:2545:server_fill_super()) Unable to start targets: -2 [ 1507.051545] LustreError: 67217:0:(tgt_mount.c:2063:server_put_super()) no obd lustre-MDT0000 [ 1507.054383] LustreError: 67217:0:(tgt_mount.c:114:server_deregister_mount()) lustre-MDT0000 not registered [ 1507.067375] Lustre: server umount lustre-MDT0000 complete [ 1507.069413] LustreError: 67217:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1509.661388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1509.712229] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1509.827879] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1509.863693] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1511.021698] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1511.361427] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1514.982838] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1514.993051] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 1515.009197] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1516.511938] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1516.515598] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1516.521061] Lustre: Skipped 1 previous similar message [ 1516.523262] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1520.096748] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1520.097874] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1520.102453] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1522.130941] Lustre: server umount lustre-OST0000 complete [ 1525.216930] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1525.217288] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1525.217655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1525.217659] Lustre: Skipped 1 previous similar message [ 1525.222355] Lustre: Skipped 1 previous similar message [ 1529.695210] Lustre: server umount lustre-MDT0000 complete [ 1530.337133] LustreError: 64458:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1530.344878] LustreError: 64458:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1531.017863] LustreError: 64439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775885565 with bad export cookie 11740237086261130878 [ 1531.020284] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1531.021368] LustreError: 64439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1531.144617] Lustre: server umount lustre-MDT0001 complete [ 1535.197936] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 1536.314272] Key type lgssc unregistered [ 1536.463694] LNet: 68972:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1536.466208] LNetError: 68972:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1536.479284] LNet: Removed LNI 192.168.202.131@tcp [ 1536.804991] Key type .llcrypt unregistered [ 1536.806491] Key type ._llcrypt unregistered [ 1546.918739] Key type ._llcrypt registered [ 1546.921046] Key type .llcrypt registered [ 1546.963305] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 1547.461321] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1547.470315] alg: No test for adler32 (adler32-zlib) [ 1548.351638] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 1548.445823] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 1550.039164] Key type lgssc registered [ 1550.440195] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1553.678967] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1553.685985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1554.874903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1556.500829] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1559.855347] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1560.014440] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1561.587701] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1563.813520] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1565.189601] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1567.741647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1567.875539] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1569.919496] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1572.009785] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1580.203314] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 1585.120848] Lustre: Failing over lustre-MDT0000 [ 1585.301514] Lustre: server umount lustre-MDT0000 complete [ 1585.637281] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1585.639127] LustreError: 70344:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1585.642806] Lustre: Skipped 2 previous similar messages [ 1585.649467] LustreError: 70344:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1585.655826] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1587.834935] LustreError: 70339:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1587.840983] LustreError: 70339:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1589.079731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1589.126173] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1589.187066] Lustre: *** cfs_fail_loc=a05, val=0*** [ 1589.244234] LustreError: 73110:0:(obd_config.c:843:class_setup()) setup lustre-MDT0000 failed (-9) [ 1589.246757] LustreError: 73110:0:(obd_config.c:2101:class_config_llog_handler()) MGC192.168.202.131@tcp: cfg command failed: rc = -9 [ 1589.249459] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1589.249459] [ 1589.252919] LustreError: MGC192.168.202.131@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1589.256907] LustreError: 73099:0:(tgt_mount.c:1819:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 1589.259745] LustreError: 73099:0:(tgt_mount.c:2545:server_fill_super()) Unable to start targets: -9 [ 1589.262777] LustreError: 73099:0:(tgt_mount.c:2063:server_put_super()) no obd lustre-MDT0000 [ 1589.265029] LustreError: 73099:0:(tgt_mount.c:114:server_deregister_mount()) lustre-MDT0000 not registered [ 1589.276987] Lustre: server umount lustre-MDT0000 complete [ 1589.278681] LustreError: 73099:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 1590.753482] LustreError: 71954:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1590.759289] LustreError: 71954:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1591.771701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1591.809662] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1591.908921] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1591.941756] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1592.943404] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1593.348889] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1597.412595] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1597.427258] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 1597.445740] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 1600.991817] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1600.994148] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1600.998652] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1602.530506] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1602.531548] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1602.536912] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1604.625453] Lustre: server umount lustre-OST0000 complete [ 1607.649031] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1607.649831] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1607.659786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1607.662036] Lustre: Skipped 2 previous similar messages [ 1612.182209] Lustre: server umount lustre-MDT0000 complete [ 1612.769323] LustreError: 71053:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1612.774927] LustreError: 71053:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1613.676278] LustreError: 74125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775885647 with bad export cookie 6232317750846697760 [ 1613.680134] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1613.687334] LustreError: 74125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1613.802282] Lustre: server umount lustre-MDT0001 complete [ 1618.911892] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 1620.216195] Key type lgssc unregistered [ 1620.362401] LNet: 74858:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1620.365359] LNetError: 74858:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1620.374438] LNet: Removed LNI 192.168.202.131@tcp [ 1620.729700] Key type .llcrypt unregistered [ 1620.731048] Key type ._llcrypt unregistered [ 1625.353379] Key type ._llcrypt registered [ 1625.354484] Key type .llcrypt registered [ 1625.393799] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 1625.503399] Key type .llcrypt unregistered [ 1625.504645] Key type ._llcrypt unregistered [ 1635.366947] Key type ._llcrypt registered [ 1635.368833] Key type .llcrypt registered [ 1635.409288] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 1635.847719] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1635.856840] alg: No test for adler32 (adler32-zlib) [ 1636.728727] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 1636.819638] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 1638.416164] Key type lgssc registered [ 1638.887996] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1641.986930] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1641.996939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1643.184608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1644.727846] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1648.168187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1648.311649] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1649.768191] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1652.099677] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1653.631224] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1656.336832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1656.491403] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1658.744592] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1660.890452] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1668.779352] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 1679.329346] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1679.331213] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1679.332185] LustreError: Skipped 1 previous similar message [ 1679.338235] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1679.340799] Lustre: Skipped 1 previous similar message [ 1680.817632] Lustre: server umount lustre-OST0000 complete [ 1684.450262] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1684.451169] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1684.451670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1684.455394] Lustre: Skipped 2 previous similar messages [ 1689.570190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1689.573229] Lustre: Skipped 1 previous similar message [ 1694.690591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1694.693196] Lustre: Skipped 1 previous similar message [ 1697.247160] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1697.320644] Lustre: server umount lustre-MDT0000 complete [ 1698.723355] LustreError: 77418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775885732 with bad export cookie 7072182608526715849 [ 1698.726040] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1698.728968] LustreError: 77418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1698.840744] Lustre: server umount lustre-MDT0001 complete [ 1702.856879] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 1703.983290] Key type lgssc unregistered [ 1704.122437] LNet: 79902:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1704.125859] LNetError: 79902:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1704.134364] LNet: Removed LNI 192.168.202.131@tcp [ 1704.473362] Key type .llcrypt unregistered [ 1704.474856] Key type ._llcrypt unregistered [ 1714.187133] Key type ._llcrypt registered [ 1714.188354] Key type .llcrypt registered [ 1714.247182] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 1714.670099] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1714.728587] alg: No test for adler32 (adler32-zlib) [ 1715.621216] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 1715.737168] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 1717.327184] Key type lgssc registered [ 1717.694836] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1720.441736] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1720.449069] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1721.634836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1723.177035] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1726.643531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1726.800105] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1728.260222] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1730.410458] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1731.987127] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1734.490419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1734.602233] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1736.602647] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1738.646043] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1745.892786] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 1752.033197] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1752.036251] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1752.046157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1756.130191] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1756.130827] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1756.131065] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1761.249880] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1761.252101] Lustre: Skipped 1 previous similar message [ 1766.367109] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1766.370112] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1766.372385] Lustre: Skipped 1 previous similar message [ 1766.421585] Lustre: server umount lustre-OST0000 complete [ 1767.905873] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1767.909273] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1771.489674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1771.490217] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1771.492077] Lustre: Skipped 2 previous similar messages [ 1781.728648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1781.731288] Lustre: Skipped 4 previous similar messages [ 1782.239172] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1782.307322] Lustre: server umount lustre-MDT0000 complete [ 1783.814299] LustreError: 81255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775885818 with bad export cookie 17281718666773876065 [ 1783.817655] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1783.819448] LustreError: 81255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1783.946481] Lustre: server umount lustre-MDT0001 complete [ 1787.854579] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 1788.985774] Key type lgssc unregistered [ 1789.123599] LNet: 84473:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1789.130541] LNetError: 84473:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1789.140078] LNet: Removed LNI 192.168.202.131@tcp [ 1789.483119] Key type .llcrypt unregistered [ 1789.485171] Key type ._llcrypt unregistered [ 1798.880794] Key type ._llcrypt registered [ 1798.881773] Key type .llcrypt registered [ 1798.928100] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 1799.383111] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1799.421987] alg: No test for adler32 (adler32-zlib) [ 1800.282963] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 1800.369900] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 1801.959209] Key type lgssc registered [ 1802.414245] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1805.384621] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1805.393669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1806.614992] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1808.139727] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1811.569801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1811.814266] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1813.513301] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1815.557418] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1816.901855] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1819.619191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1819.835455] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1821.976036] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1823.978181] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1831.078724] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 1841.635037] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1841.635270] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1841.642719] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1841.645523] Lustre: Skipped 1 previous similar message [ 1844.813704] Lustre: server umount lustre-OST0000 complete [ 1846.752161] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1846.753797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1846.757152] Lustre: Skipped 2 previous similar messages [ 1851.872135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1851.873944] Lustre: Skipped 2 previous similar messages [ 1852.307926] Lustre: server umount lustre-MDT0000 complete [ 1853.491807] LustreError: 88008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775885887 with bad export cookie 8208452897529205622 [ 1853.495082] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1853.497116] LustreError: 88008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1853.619429] Lustre: server umount lustre-MDT0001 complete [ 1857.206426] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 1858.248248] Key type lgssc unregistered [ 1858.383364] LNet: 89044:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1858.385686] LNetError: 89044:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1858.393370] LNet: Removed LNI 192.168.202.131@tcp [ 1858.699507] Key type .llcrypt unregistered [ 1858.701283] Key type ._llcrypt unregistered [ 1867.871542] Key type ._llcrypt registered [ 1867.873120] Key type .llcrypt registered [ 1867.915616] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 1868.404766] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1868.425924] alg: No test for adler32 (adler32-zlib) [ 1869.275934] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 1869.366523] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 1870.951189] Key type lgssc registered [ 1871.368508] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1874.259841] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1874.266444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1889.695253] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1894.816071] LustreError: 90390:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.131@tcp: failed processing log, type 4: rc = -110 [ 1921.439187] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1923.450831] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1925.405470] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1927.425886] Lustre: server umount lustre-OST0000 complete [ 1930.251400] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 01:39:24 (1775885964) [ 1932.345052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1932.472850] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1933.857458] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1936.869281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1936.989669] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1938.259302] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1940.124442] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1941.370516] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1943.576043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1943.690150] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1945.511530] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1947.335776] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1957.031530] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 1962.469460] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1962.469924] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1962.470725] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1962.473453] LustreError: Skipped 1 previous similar message [ 1962.477492] Lustre: Skipped 1 previous similar message [ 1963.532822] Lustre: server umount lustre-OST0000 complete [ 1967.585200] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1967.586072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1967.590435] Lustre: Skipped 1 previous similar message [ 1967.592387] Lustre: Skipped 1 previous similar message [ 1967.594177] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1970.965992] Lustre: server umount lustre-MDT0000 complete [ 1972.304490] LustreError: 92630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775886006 with bad export cookie 4511495911212019667 [ 1972.307870] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1972.308843] LustreError: 92630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1972.417637] Lustre: server umount lustre-MDT0001 complete [ 1975.758235] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 1976.828951] Key type lgssc unregistered [ 1976.959572] LNet: 95269:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1976.963174] LNetError: 95269:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1976.970487] LNet: Removed LNI 192.168.202.131@tcp [ 1977.316882] Key type .llcrypt unregistered [ 1977.318096] Key type ._llcrypt unregistered [ 1985.871963] Key type ._llcrypt registered [ 1985.873586] Key type .llcrypt registered [ 1985.912645] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 1986.211444] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1986.249723] alg: No test for adler32 (adler32-zlib) [ 1987.093703] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 1987.171643] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 1988.751120] Key type lgssc registered [ 1989.064239] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1991.620610] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1991.626232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1992.774426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1994.038551] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 1996.826901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1996.951895] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1998.313750] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2000.090127] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2001.350598] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2003.798473] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2003.954648] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2005.893886] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2007.908313] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2017.257128] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 2022.368586] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2022.371993] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2022.374140] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2022.378476] Lustre: Skipped 1 previous similar message [ 2022.991782] Lustre: server umount lustre-OST0000 complete [ 2026.318734] Lustre: server umount lustre-MDT0000 complete [ 2027.491422] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2027.492168] LustreError: 98789:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2027.496323] Lustre: Skipped 2 previous similar messages [ 2027.503092] LustreError: 98789:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2027.521027] LustreError: 96623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775886061 with bad export cookie 1300013431030083735 [ 2027.524289] LustreError: 96623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2027.524297] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2027.646373] Lustre: server umount lustre-MDT0001 complete [ 2031.198132] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 2032.195660] Key type lgssc unregistered [ 2032.327380] LNet: 99838:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2032.329420] LNetError: 99838:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2032.340297] LNet: Removed LNI 192.168.202.131@tcp [ 2032.638333] Key type .llcrypt unregistered [ 2032.639479] Key type ._llcrypt unregistered [ 2040.241336] Key type ._llcrypt registered [ 2040.242323] Key type .llcrypt registered [ 2040.286303] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 2040.687441] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2040.699721] alg: No test for adler32 (adler32-zlib) [ 2041.557226] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 2041.634281] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 2043.215155] Key type lgssc registered [ 2043.594560] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2046.281970] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2046.288797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2061.727331] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2066.847234] LustreError: 101041:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.131@tcp: failed processing log, type 4: rc = -110 [ 2093.471142] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2095.389020] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2097.242775] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2099.579901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2099.674653] LustreError: MGC192.168.202.131@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2099.680104] LustreError: 101949:0:(tgt_mount.c:1819:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2099.683430] LustreError: 101949:0:(tgt_mount.c:2545:server_fill_super()) Unable to start targets: -2 [ 2099.687383] LustreError: 101949:0:(tgt_mount.c:2063:server_put_super()) no obd lustre-MDT0000 [ 2099.727636] Lustre: server umount lustre-MDT0000 complete [ 2099.729042] LustreError: 101949:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2112.002299] Lustre: server umount lustre-OST0000 complete [ 2115.107598] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_hostid [ 2118.428870] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 2130.693297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2130.783288] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2130.795930] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2130.833318] Lustre: lustre-MDT0000: new disk, initializing [ 2130.860655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2130.868751] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2132.099861] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2136.362323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2136.404600] Lustre: 104699:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2136.429358] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2136.432033] Lustre: Skipped 1 previous similar message [ 2136.467578] Lustre: lustre-MDT0001: new disk, initializing [ 2136.498432] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2136.509733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2136.514153] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2137.902530] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2141.166582] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2142.476493] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2144.780084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2144.878281] Lustre: lustre-OST0000: new disk, initializing [ 2144.880412] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2144.906677] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2146.686139] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2147.323658] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2147.327034] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2147.356897] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2149.841617] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2151.341409] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2151.419098] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2152.679140] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2152.742017] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2157.538472] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2157.538492] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2157.539684] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2157.541341] LustreError: Skipped 1 previous similar message [ 2157.545452] Lustre: Skipped 1 previous similar message [ 2159.435044] Lustre: server umount lustre-OST0000 complete [ 2162.144091] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2162.146860] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2162.151869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2162.153657] Lustre: Skipped 1 previous similar message [ 2166.808215] Lustre: server umount lustre-MDT0000 complete [ 2168.014558] LustreError: 106440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775886202 with bad export cookie 8229154139971060938 [ 2168.017396] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2168.018116] LustreError: 106440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2168.141773] Lustre: server umount lustre-MDT0001 complete [ 2170.910824] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 01:43:24 (1775886204) [ 2171.949541] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 2176.945549] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_hostid [ 2179.926665] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 2191.522361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2191.606507] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2191.616804] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2191.652197] Lustre: lustre-MDT0000: new disk, initializing [ 2191.674766] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2191.680915] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2192.973087] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2196.972658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2197.011064] Lustre: 111150:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2197.032557] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2197.035018] Lustre: Skipped 1 previous similar message [ 2197.069886] Lustre: lustre-MDT0001: new disk, initializing [ 2197.113799] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2197.116790] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2198.341060] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2201.455515] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2202.590273] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2204.919285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2205.052107] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2205.054700] Lustre: Skipped 1 previous similar message [ 2206.934761] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2209.272372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2209.277161] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2209.308029] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2210.134911] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2211.504519] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2211.576181] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2212.883965] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2212.958993] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2214.369853] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2214.370213] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2214.371488] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2214.371493] Lustre: Skipped 1 previous similar message [ 2214.372576] LustreError: Skipped 1 previous similar message [ 2214.376149] Lustre: Skipped 2 previous similar messages [ 2215.699715] Lustre: server umount lustre-OST0000 complete [ 2217.951903] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2217.955599] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2217.960270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2217.962167] Lustre: Skipped 1 previous similar message [ 2224.607931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2224.610052] Lustre: Skipped 3 previous similar messages [ 2231.263100] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2231.321493] Lustre: server umount lustre-MDT0000 complete [ 2232.601741] LustreError: 111143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775886266 with bad export cookie 8229154139971061806 [ 2232.603803] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2232.606986] LustreError: 111143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2232.737423] Lustre: server umount lustre-MDT0001 complete [ 2235.928955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2236.072060] LustreError: 114958:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2236.078327] LustreError: 114958:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2236.097438] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2237.499467] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2240.696058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2242.173136] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2244.015213] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2245.317548] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2247.724025] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2249.716646] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2251.618029] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2264.556082] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 2266.593130] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2266.595503] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2266.596920] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2266.600704] Lustre: Skipped 2 previous similar messages [ 2266.604581] Lustre: Skipped 3 previous similar messages [ 2267.536949] Lustre: server umount lustre-OST0000 complete [ 2274.971402] Lustre: server umount lustre-MDT0000 complete [ 2276.308915] LustreError: 117233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775886310 with bad export cookie 8229154139971062674 [ 2276.312126] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2276.313561] LustreError: 117233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2276.423779] Lustre: server umount lustre-MDT0001 complete [ 2280.228315] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 2281.268329] Key type lgssc unregistered [ 2281.407316] LNet: 118253:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2281.410658] LNetError: 118253:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2281.418283] LNet: Removed LNI 192.168.202.131@tcp [ 2281.721809] Key type .llcrypt unregistered [ 2281.723079] Key type ._llcrypt unregistered [ 2287.674763] Key type ._llcrypt registered [ 2287.675610] Key type .llcrypt registered [ 2287.711373] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_hostid [ 2294.321092] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 2294.805278] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2294.825145] alg: No test for adler32 (adler32-zlib) [ 2295.695344] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 2295.786872] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 2297.375214] Key type lgssc registered [ 2297.802923] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2310.373315] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2310.382993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2311.470944] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2311.484265] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2311.522353] Lustre: lustre-MDT0000: new disk, initializing [ 2311.560700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2311.566743] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2312.908529] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2317.123179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2317.161261] Lustre: 121344:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2317.188525] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2317.191092] Lustre: Skipped 1 previous similar message [ 2317.225578] Lustre: lustre-MDT0001: new disk, initializing [ 2317.263119] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2317.277314] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2317.280913] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2318.676384] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2321.915709] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2323.170709] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2325.349460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2325.454054] Lustre: lustre-OST0000: new disk, initializing [ 2325.456819] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2325.498939] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2327.301847] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2330.390444] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2330.600464] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2330.603901] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2330.619669] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2331.813616] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2331.883692] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2333.165437] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2333.240270] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2335.712660] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2335.713376] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2335.714934] LustreError: Skipped 1 previous similar message [ 2335.719027] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2335.721953] Lustre: Skipped 1 previous similar message [ 2335.950980] Lustre: server umount lustre-OST0000 complete [ 2337.759860] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2337.762104] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2337.767597] Lustre: Skipped 1 previous similar message [ 2337.769701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2340.832148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2340.832889] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2343.321531] Lustre: server umount lustre-MDT0000 complete [ 2344.566333] LustreError: 121336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775886378 with bad export cookie 17553096989735133681 [ 2344.568870] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2344.570867] LustreError: 121336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2344.693713] Lustre: server umount lustre-MDT0001 complete [ 2347.781916] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 01:46:21 (1775886381) [ 2350.198909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2350.393293] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2351.911707] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2355.041226] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2355.188305] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2356.488289] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2358.256033] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2359.516989] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2360.223685] LustreError: 125377:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2360.226125] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2360.226528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2360.226532] Lustre: Skipped 1 previous similar message [ 2360.227782] LustreError: 125377:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2360.232019] Lustre: Skipped 1 previous similar message [ 2360.241272] LustreError: 125377:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 10, retries 0, failed: rc = -5 [ 2365.344446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2365.346347] Lustre: Skipped 2 previous similar messages [ 2366.425594] Lustre: server umount lustre-MDT0000 complete [ 2367.658837] LustreError: 125330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775886401 with bad export cookie 17553096989735134549 [ 2367.661956] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2367.662912] LustreError: 125330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2367.765753] Lustre: server umount lustre-MDT0001 complete [ 2370.495324] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 01:46:44 (1775886404) [ 2372.740859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2388.191592] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2393.311463] LustreError: 127533:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.131@tcp: failed processing log, type 4: rc = -110 [ 2418.911266] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2420.952182] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2422.963704] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2423.735372] Lustre: server umount lustre-OST0000 complete [ 2426.415719] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 01:47:40 (1775886460) [ 2428.756352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2428.911220] LustreError: 128862:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2428.919594] LustreError: 128862:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2430.250337] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2433.090021] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2434.541213] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2436.476522] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2437.777097] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2440.126872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2440.239250] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2440.241260] Lustre: Skipped 2 previous similar messages [ 2442.064434] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2443.805078] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2464.223890] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2464.227879] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2464.234448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2466.907595] Lustre: server umount lustre-MDT0000 complete [ 2468.252143] LustreError: 129580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775886502 with bad export cookie 17553096989735135179 [ 2468.254727] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2468.255925] LustreError: 129580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2468.364658] Lustre: server umount lustre-MDT0001 complete [ 2478.938723] Lustre: server umount lustre-OST0000 complete [ 2481.782478] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 01:48:35 (1775886515) [ 2484.244089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2484.386671] LustreError: 132198:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2484.410232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2485.708847] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2488.791179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2490.332589] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2492.183113] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2493.527768] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2495.789906] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2497.817820] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2499.674217] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2501.152408] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2509.222884] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 2509.399710] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec [ 2511.131097] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2511.218519] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2514.401960] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2514.402354] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2514.403293] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2514.403300] Lustre: Skipped 2 previous similar messages [ 2514.407802] Lustre: Skipped 3 previous similar messages [ 2518.036136] Lustre: server umount lustre-OST0000 complete [ 2519.521098] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2519.524051] LustreError: Skipped 1 previous similar message [ 2525.530352] Lustre: server umount lustre-MDT0000 complete [ 2526.846217] LustreError: 132178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775886561 with bad export cookie 17553096989735136236 [ 2526.849357] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2526.852309] LustreError: 132178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2526.971853] Lustre: server umount lustre-MDT0001 complete [ 2529.856276] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 01:49:23 (1775886563) [ 2532.055710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2547.487638] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2552.607441] LustreError: 136037:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.131@tcp: failed processing log, type 4: rc = -110 [ 2578.207365] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2578.209796] Lustre: Skipped 2 previous similar messages [ 2580.197039] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2582.074256] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2584.372769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2584.512664] LustreError: 136062:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2584.554948] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 2585.789710] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2588.759159] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2590.200226] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2592.116041] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2593.459656] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2594.894558] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2594.976690] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2596.346475] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2596.421737] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2598.369786] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2598.371489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2598.374033] Lustre: Skipped 4 previous similar messages [ 2598.375926] Lustre: Skipped 6 previous similar messages [ 2603.295952] Lustre: server umount lustre-MDT0000 complete [ 2603.489477] LustreError: 136960:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2603.495691] LustreError: 136960:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2604.639182] LustreError: 136044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775886638 with bad export cookie 17553096989735137244 [ 2604.641427] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2604.642572] LustreError: 136044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2604.755507] Lustre: server umount lustre-MDT0001 complete [ 2616.415139] Lustre: server umount lustre-OST0000 complete [ 2620.618419] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 01:50:54 (1775886654) [ 2624.098303] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2639.583561] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2644.703427] LustreError: 139859:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.131@tcp: failed processing log, type 4: rc = -110 [ 2672.327356] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2674.283665] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2676.704452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2676.863206] LustreError: 139884:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2676.939148] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 2678.323456] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2681.318513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2682.832117] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2684.708992] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2685.959824] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2688.338848] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2688.409498] Lustre: lustre-OST0001: new disk, initializing [ 2688.411912] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2690.249479] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2693.514680] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2694.913896] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2695.670493] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2695.673840] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2695.702793] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2695.996616] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 2697.290052] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2697.366367] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2700.771685] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2700.772282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2700.775778] Lustre: Skipped 1 previous similar message [ 2704.054371] Lustre: server umount lustre-OST0000 complete [ 2705.889146] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2705.890693] LustreError: 139875:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2705.892228] LustreError: Skipped 1 previous similar message [ 2705.900085] LustreError: 139875:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2711.375726] Lustre: server umount lustre-OST0001 complete [ 2718.675550] Lustre: server umount lustre-MDT0000 complete [ 2719.941111] LustreError: 141493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775886754 with bad export cookie 17553096989735138196 [ 2719.943813] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2719.945632] LustreError: 141493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2720.063736] Lustre: server umount lustre-MDT0001 complete [ 2727.840929] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 01:52:41 (1775886761) [ 2728.439032] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 2729.007175] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 2729.605251] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 01:52:43 (1775886763) [ 2731.994119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2732.100689] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2732.113400] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2732.183352] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2732.185267] Lustre: Skipped 6 previous similar messages [ 2733.582995] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2737.101309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2737.157912] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2737.169235] Lustre: 145809:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2738.714351] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2740.822701] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2742.255213] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2744.873042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2744.953066] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2747.046573] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2748.690484] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 2749.042777] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2750.641456] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2750.714283] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2752.147198] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2752.219146] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2754.016694] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2754.019213] LustreError: Skipped 1 previous similar message [ 2758.993279] Lustre: server umount lustre-OST0000 complete [ 2772.954410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2774.116558] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2774.120798] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2774.120846] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2774.128078] Lustre: Skipped 1 previous similar message [ 2775.123109] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2777.070227] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2778.672492] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2778.747935] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2780.037322] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2780.106059] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2781.339226] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff891c8822a000.ost_server_uuid 50 [ 2781.850204] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff891c8822a000.ost_server_uuid in FULL state after 0 sec [ 2781.908915] LustreError: 145815:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.202.31@tcp) returned error from blocking AST (req@00000000f6ed90a5 x1862151784549376 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 000000008a429767/0xf39920b63d243ce0 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.31@tcp remote: 0x22a3a926225c1a9b expref: 5 pid: 145816 timeout: 2881 lvb_type: 0 lru_score: 0 lru_type: 0 [ 2781.919285] LustreError: lustre-MDT0000: A client on nid 192.168.202.31@tcp was evicted due to a lock blocking callback time out: rc -107 [ 2781.922348] LustreError: 145804:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.202.31@tcp ns: mdt-lustre-MDT0000_UUID lock: 000000008a429767/0xf39920b63d243ce0 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.31@tcp remote: 0x22a3a926225c1a9b expref: 6 pid: 145816 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 2784.738307] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2784.742037] Lustre: Skipped 7 previous similar messages [ 2789.131948] Lustre: server umount lustre-OST0000 complete [ 2793.953158] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2793.955791] LustreError: Skipped 2 previous similar messages [ 2796.502680] Lustre: server umount lustre-MDT0000 complete [ 2797.814465] LustreError: 147442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775886832 with bad export cookie 17553096989735139379 [ 2797.818043] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2797.818833] LustreError: 147442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2812.383101] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2812.483106] Lustre: server umount lustre-MDT0001 complete [ 2815.881975] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 2816.855653] Key type lgssc unregistered [ 2816.976409] LNet: 151281:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2816.978421] LNetError: 151281:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2816.990294] LNet: Removed LNI 192.168.202.131@tcp [ 2817.264591] Key type .llcrypt unregistered [ 2817.266058] Key type ._llcrypt unregistered [ 2826.041425] Key type ._llcrypt registered [ 2826.042677] Key type .llcrypt registered [ 2826.087053] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 2826.499462] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2826.507379] alg: No test for adler32 (adler32-zlib) [ 2827.353036] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 2827.432838] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 2829.015230] Key type lgssc registered [ 2829.389520] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2831.846168] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2831.851748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2832.992634] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2834.230344] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2836.957694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2837.072856] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2838.357417] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2840.041579] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2841.207939] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2843.294166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2843.390660] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2845.078259] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2846.701414] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2858.788989] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 2864.262176] Lustre: Failing over lustre-MDT0000 [ 2864.482978] Lustre: server umount lustre-MDT0000 complete [ 2867.679752] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2867.682160] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2867.685988] LustreError: 152653:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2867.690552] LustreError: 152653:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2869.216461] LustreError: 153375:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2869.217268] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2869.221103] LustreError: 153375:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2869.228501] Lustre: Skipped 1 previous similar message [ 2874.336724] LustreError: 152654:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2874.341151] LustreError: 152654:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2875.377180] LustreError: 152634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.31@tcp arrived at 1775886909 with bad export cookie 2066013354225012428 [ 2875.381910] LustreError: 152634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2877.858973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2877.898791] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2877.989065] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2878.013425] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2879.211820] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2879.918662] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2879.921547] Lustre: lustre-MDT0000: Denying connection for new client 19f0646d-3160-4f62-8859-dae64339e4ce (at 192.168.202.31@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2883.044212] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2888.160587] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2888.162237] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2888.163835] Lustre: Skipped 1 previous similar message [ 2892.747471] Lustre: server umount lustre-OST0000 complete [ 2893.907115] LustreError: 156049:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2893.909864] Lustre: 155416:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2893.912326] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2893.917827] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2893.921615] LustreError: 155416:0:(client.c:1383:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000003459126c x1862152341195392/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2893.926613] LustreError: 155416:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 2893.929404] LustreError: 155416:0:(fid_request.c:316:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 2893.932287] Lustre: lustre-MDT0000: Recovery over after 0:14, of 2 clients 0 recovered and 2 were evicted. [ 2894.022329] Lustre: server umount lustre-MDT0000 complete [ 2895.142282] LustreError: 152635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775886929 with bad export cookie 2066013354225012785 [ 2895.145402] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2895.146516] LustreError: 152635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2895.149623] LustreError: 151964:0:(import.c:706:ptlrpc_connect_import_locked()) can't connect to a closed import [ 2901.293545] Lustre: server umount lustre-MDT0001 complete [ 2904.709095] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 2905.775628] Key type lgssc unregistered [ 2905.900493] LNet: 156784:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2905.903266] LNetError: 156784:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2905.916603] LNet: Removed LNI 192.168.202.131@tcp [ 2906.210144] Key type .llcrypt unregistered [ 2906.211271] Key type ._llcrypt unregistered [ 2914.966545] Key type ._llcrypt registered [ 2914.967670] Key type .llcrypt registered [ 2915.003791] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 2915.316308] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2915.361320] alg: No test for adler32 (adler32-zlib) [ 2916.226091] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 2916.308095] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 2917.887153] Key type lgssc registered [ 2918.218577] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2920.719580] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2920.728758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2921.901293] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2923.157851] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2925.863299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2925.985569] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2927.309758] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2929.314448] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2930.566392] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2933.009456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2933.134672] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2935.112726] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2936.955067] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2947.495777] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 2951.648568] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2951.652373] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2951.659266] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2952.672945] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2952.673524] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2952.674810] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2955.028816] Lustre: server umount lustre-OST0000 complete [ 2956.767847] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2956.770966] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2956.776142] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2956.778407] Lustre: Skipped 1 previous similar message [ 2962.457688] Lustre: server umount lustre-MDT0000 complete [ 2962.912502] LustreError: 158157:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2962.918856] LustreError: 158157:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2963.776587] LustreError: 158137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775886997 with bad export cookie 1483968791743589426 [ 2963.779342] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2963.781110] LustreError: 158137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2963.894553] Lustre: server umount lustre-MDT0001 complete [ 2967.657208] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 2968.737951] Key type lgssc unregistered [ 2968.871373] LNet: 161356:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2968.873390] LNetError: 161356:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2968.880393] LNet: Removed LNI 192.168.202.131@tcp [ 2969.169687] Key type .llcrypt unregistered [ 2969.170656] Key type ._llcrypt unregistered [ 2981.281052] Key type ._llcrypt registered [ 2981.282023] Key type .llcrypt registered [ 2981.320300] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 2981.707389] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2981.720410] alg: No test for adler32 (adler32-zlib) [ 2982.574604] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 2982.660141] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 2984.239166] Key type lgssc registered [ 2984.623776] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2987.229871] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2987.236516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2988.371053] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2989.502345] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2992.571605] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2992.701471] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2993.999454] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 2995.834489] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2997.072740] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2999.391502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2999.499346] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3001.234718] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3002.969913] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3012.839283] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 3018.210690] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3018.211154] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3018.212036] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3018.214065] Lustre: Skipped 1 previous similar message [ 3021.198508] Lustre: server umount lustre-OST0000 complete [ 3023.328541] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3023.328625] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3023.329633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3023.329638] Lustre: Skipped 1 previous similar message [ 3023.330724] LustreError: Skipped 1 previous similar message [ 3023.333759] Lustre: Skipped 1 previous similar message [ 3028.448113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3028.449718] Lustre: Skipped 2 previous similar messages [ 3028.563879] Lustre: server umount lustre-MDT0000 complete [ 3029.875841] LustreError: 162997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775887064 with bad export cookie 17268633585802797941 [ 3029.878262] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3029.879252] LustreError: 162997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3029.984303] Lustre: server umount lustre-MDT0001 complete [ 3033.366141] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 3034.289656] Key type lgssc unregistered [ 3034.408372] LNet: 166215:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3034.410850] LNetError: 166215:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3034.420287] LNet: Removed LNI 192.168.202.131@tcp [ 3034.698688] Key type .llcrypt unregistered [ 3034.699584] Key type ._llcrypt unregistered [ 3043.198176] Key type ._llcrypt registered [ 3043.199108] Key type .llcrypt registered [ 3043.230333] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 3043.556280] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3043.575678] alg: No test for adler32 (adler32-zlib) [ 3044.418326] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 3044.490816] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 3046.071202] Key type lgssc registered [ 3046.490058] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3050.315503] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3050.322367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3051.451221] Lustre: *** cfs_fail_loc=135, val=0*** [ 3051.491381] LustreError: 167697:0:(obd_config.c:843:class_setup()) setup lustre-MDT0000 failed (-2) [ 3051.494904] LustreError: 167697:0:(obd_config.c:2101:class_config_llog_handler()) MGC192.168.202.131@tcp: cfg command failed: rc = -2 [ 3051.499184] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 3051.499184] [ 3051.503666] LustreError: MGC192.168.202.131@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3051.507970] LustreError: 167658:0:(tgt_mount.c:1819:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3051.511171] LustreError: 167658:0:(tgt_mount.c:2545:server_fill_super()) Unable to start targets: -2 [ 3051.513243] LustreError: 167658:0:(tgt_mount.c:2063:server_put_super()) no obd lustre-MDT0000 [ 3051.514953] LustreError: 167658:0:(tgt_mount.c:114:server_deregister_mount()) lustre-MDT0000 not registered [ 3051.561932] Lustre: server umount lustre-MDT0000 complete [ 3051.563427] LustreError: 167658:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3053.868781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3054.026813] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3055.320335] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3057.914725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3058.019323] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3059.258522] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3060.931730] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3062.130557] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3062.797566] LustreError: 168104:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3062.801240] LustreError: 168104:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3062.805308] LustreError: 168104:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -5 [ 3063.265051] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3063.265325] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3063.265756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3063.268515] Lustre: Skipped 1 previous similar message [ 3068.384247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3068.386031] Lustre: Skipped 2 previous similar messages [ 3073.504261] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3073.506591] Lustre: Skipped 1 previous similar message [ 3077.087117] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3077.145208] Lustre: server umount lustre-MDT0000 complete [ 3078.539173] LustreError: 168055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775887112 with bad export cookie 13397780400748406130 [ 3078.542203] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3078.543405] LustreError: 168055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3078.649142] Lustre: server umount lustre-MDT0001 complete [ 3082.087203] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 3083.065884] Key type lgssc unregistered [ 3083.192320] LNet: 170152:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3083.194866] LNetError: 170152:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3083.204303] LNet: Removed LNI 192.168.202.131@tcp [ 3083.456315] Key type .llcrypt unregistered [ 3083.457760] Key type ._llcrypt unregistered [ 3090.026023] Key type ._llcrypt registered [ 3090.026877] Key type .llcrypt registered [ 3090.057037] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 3090.141188] Key type .llcrypt unregistered [ 3090.142054] Key type ._llcrypt unregistered [ 3097.172800] Key type ._llcrypt registered [ 3097.173546] Key type .llcrypt registered [ 3097.204079] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 3097.499516] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3097.546827] alg: No test for adler32 (adler32-zlib) [ 3098.412435] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 3098.498509] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 3100.087161] Key type lgssc registered [ 3100.479058] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3103.232620] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3103.238863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3118.687398] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3123.807152] LustreError: 172224:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.131@tcp: failed processing log, type 4: rc = -110 [ 3150.431112] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3152.254649] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3153.885741] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3155.914470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3156.055462] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3156.071573] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 3157.281425] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3160.138249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3160.271048] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3161.601502] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3163.312434] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3164.479664] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3165.434630] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 3175.906850] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3175.907490] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3175.908711] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3175.912870] Lustre: Skipped 1 previous similar message [ 3177.613745] Lustre: server umount lustre-OST0000 complete [ 3181.025134] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3181.025296] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3181.026026] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3181.026030] Lustre: Skipped 1 previous similar message [ 3181.035347] Lustre: Skipped 1 previous similar message [ 3186.144099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3186.146243] Lustre: Skipped 2 previous similar messages [ 3191.264444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3191.267660] Lustre: Skipped 1 previous similar message [ 3193.311123] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3193.365739] Lustre: server umount lustre-MDT0000 complete [ 3194.683218] LustreError: 172230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775887228 with bad export cookie 4111929368761056385 [ 3194.686850] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3194.691844] LustreError: 172230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3194.803697] Lustre: server umount lustre-MDT0001 complete [ 3198.431755] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 3199.481618] Key type lgssc unregistered [ 3199.608456] LNet: 175836:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3199.611177] LNetError: 175836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3199.618466] LNet: Removed LNI 192.168.202.131@tcp [ 3199.959811] Key type .llcrypt unregistered [ 3199.961050] Key type ._llcrypt unregistered [ 3208.957025] Key type ._llcrypt registered [ 3208.958650] Key type .llcrypt registered [ 3208.992604] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 3209.474634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3209.497592] alg: No test for adler32 (adler32-zlib) [ 3210.368414] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 3210.455260] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 3212.047173] Key type lgssc registered [ 3212.476155] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3215.230408] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3215.236100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3216.407641] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3217.727733] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3220.678812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3220.818109] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3222.108436] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3223.866379] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3225.075093] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3227.382486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3227.521541] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3229.427471] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3231.179982] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3240.870866] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 3242.758916] Lustre: Failing over lustre-MDT0000 [ 3242.849173] Lustre: server umount lustre-MDT0000 complete [ 3246.048731] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3246.050400] LustreError: 177204:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3246.052803] Lustre: Skipped 1 previous similar message [ 3246.059311] LustreError: 177204:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3246.705261] LustreError: 177204:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3251.168168] LustreError: 177208:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3251.173916] LustreError: 177208:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3255.713640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3255.762305] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3255.860353] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3255.882602] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3256.939799] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3257.212107] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3258.733963] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 3260.900649] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3260.908969] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 3260.924761] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 3264.579370] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 3271.140285] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3271.140522] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3271.141078] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3271.144282] Lustre: Skipped 2 previous similar messages [ 3276.257530] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3276.259527] Lustre: Skipped 2 previous similar messages [ 3276.881473] Lustre: server umount lustre-OST0000 complete [ 3281.378201] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3281.378340] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3281.379462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3281.381842] Lustre: Skipped 1 previous similar message [ 3284.250310] Lustre: server umount lustre-MDT0000 complete [ 3285.537280] LustreError: 179535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775887319 with bad export cookie 4167760831762197441 [ 3285.541531] LustreError: 179535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3285.542210] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3285.663772] Lustre: server umount lustre-MDT0001 complete [ 3289.473806] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 3290.574327] Key type lgssc unregistered [ 3290.703468] LNet: 181802:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3290.707194] LNetError: 181802:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3290.718392] LNet: Removed LNI 192.168.202.131@tcp [ 3291.044072] Key type .llcrypt unregistered [ 3291.045821] Key type ._llcrypt unregistered [ 3300.525698] Key type ._llcrypt registered [ 3300.526612] Key type .llcrypt registered [ 3300.563159] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 3300.952402] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3300.973361] alg: No test for adler32 (adler32-zlib) [ 3301.826516] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 3301.900248] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 3303.479112] Key type lgssc registered [ 3303.820473] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3306.275358] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3306.281390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3307.429534] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3308.564520] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3311.349577] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3311.461952] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3312.760635] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3314.572585] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3315.839800] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3318.075515] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3318.188408] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3319.924878] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3321.638476] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3332.516993] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 3337.881833] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 3347.470775] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 3355.451620] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3367.391741] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3367.394058] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3367.397887] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3368.417162] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3368.418581] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3368.419696] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3369.614801] Lustre: server umount lustre-OST0000 complete [ 3373.023710] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3373.026934] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3373.031771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3373.033416] Lustre: Skipped 1 previous similar message [ 3378.657224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3378.659119] Lustre: Skipped 2 previous similar messages [ 3383.776407] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3383.778644] Lustre: Skipped 1 previous similar message [ 3385.311114] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3385.367853] Lustre: server umount lustre-MDT0000 complete [ 3386.516251] LustreError: 183891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775887420 with bad export cookie 17629838656342522020 [ 3386.519607] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3386.520884] LustreError: 183891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3386.628234] Lustre: server umount lustre-MDT0001 complete [ 3390.105237] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 3391.118971] Key type lgssc unregistered [ 3391.236419] LNet: 186549:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3391.238598] LNetError: 186549:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3391.245245] LNet: Removed LNI 192.168.202.131@tcp [ 3391.497208] Key type .llcrypt unregistered [ 3391.498726] Key type ._llcrypt unregistered [ 3399.962039] Key type ._llcrypt registered [ 3399.963184] Key type .llcrypt registered [ 3399.999917] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 3400.324091] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3400.365337] alg: No test for adler32 (adler32-zlib) [ 3401.224207] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 3401.307045] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 3402.887148] Key type lgssc registered [ 3403.242461] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3405.771266] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3405.778164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3406.951279] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3408.370474] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3411.392471] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3411.513723] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3412.820246] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3414.632383] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3415.821810] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3418.108138] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3418.220385] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3419.943293] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3421.589101] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3432.553835] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 3438.714239] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 3449.024634] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 3456.854722] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 3465.599336] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 3472.865285] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3472.868182] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3472.872599] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3473.889374] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3473.891019] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3473.891121] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3478.097393] Lustre: server umount lustre-OST0000 complete [ 3483.103989] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3483.106938] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3483.111891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3483.113768] Lustre: Skipped 1 previous similar message [ 3485.530856] Lustre: server umount lustre-MDT0000 complete [ 3486.842121] LustreError: 187900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775887521 with bad export cookie 4716946310802984189 [ 3486.843758] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3486.845153] LustreError: 187900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3486.948015] Lustre: server umount lustre-MDT0001 complete [ 3490.538064] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 3491.518292] Key type lgssc unregistered [ 3491.634304] LNet: 193083:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3491.636543] LNetError: 193083:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3491.650286] LNet: Removed LNI 192.168.202.131@tcp [ 3491.929417] Key type .llcrypt unregistered [ 3491.930929] Key type ._llcrypt unregistered [ 3500.482947] Key type ._llcrypt registered [ 3500.483805] Key type .llcrypt registered [ 3500.515516] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 3500.787019] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3500.835074] alg: No test for adler32 (adler32-zlib) [ 3501.675705] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 3501.749755] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 3503.327141] Key type lgssc registered [ 3503.668109] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3506.109340] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3506.115862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3507.261845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3508.466260] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3511.210033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3511.334064] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3512.593088] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3514.231634] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3515.318972] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3517.332295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3517.433890] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3519.115501] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3520.739598] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3521.573290] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 3531.746558] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3531.748335] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3531.749235] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3531.757739] Lustre: Skipped 1 previous similar message [ 3536.273697] Lustre: server umount lustre-OST0000 complete [ 3541.985826] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3541.986061] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3541.987089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3541.987097] Lustre: Skipped 1 previous similar message [ 3541.988306] LustreError: Skipped 1 previous similar message [ 3541.992375] Lustre: Skipped 1 previous similar message [ 3543.713402] Lustre: server umount lustre-MDT0000 complete [ 3544.959147] LustreError: 196614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775887579 with bad export cookie 3644249672054674159 [ 3544.962797] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3544.962804] LustreError: 196614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3545.074709] Lustre: server umount lustre-MDT0001 complete [ 3548.692353] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 3549.690709] Key type lgssc unregistered [ 3549.818381] LNet: 197783:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3549.820668] LNetError: 197783:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3549.834296] LNet: Removed LNI 192.168.202.131@tcp [ 3550.117235] Key type .llcrypt unregistered [ 3550.118806] Key type ._llcrypt unregistered [ 3557.534929] Key type ._llcrypt registered [ 3557.536856] Key type .llcrypt registered [ 3557.573899] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 3557.977719] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3558.032705] alg: No test for adler32 (adler32-zlib) [ 3558.883110] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 3558.963197] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 3560.543140] Key type lgssc registered [ 3560.852473] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3563.562925] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3563.567676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3564.705942] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3565.818967] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3568.453725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3568.584661] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3569.762557] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3571.358757] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3572.619086] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3574.722690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3574.754849] LustreError: 200601:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3574.758349] LustreError: 200601:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3574.825096] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3576.456724] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3577.894692] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 3578.067301] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3582.945129] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3582.946766] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3582.947679] LustreError: Skipped 1 previous similar message [ 3582.948533] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3582.951579] Lustre: Skipped 1 previous similar message [ 3584.473347] Lustre: server umount lustre-OST0000 complete [ 3587.725638] Lustre: server umount lustre-MDT0000 complete [ 3588.065963] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3588.066604] LustreError: 201118:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3588.069875] Lustre: Skipped 1 previous similar message [ 3588.077582] LustreError: 201118:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3588.878983] LustreError: 201313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775887623 with bad export cookie 8468800399120121533 [ 3588.882409] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3588.882761] LustreError: 201313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3588.972690] Lustre: server umount lustre-MDT0001 complete [ 3592.243876] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 3593.181698] Key type lgssc unregistered [ 3593.295318] LNet: 202333:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3593.297741] LNetError: 202333:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3593.307281] LNet: Removed LNI 192.168.202.131@tcp [ 3593.574477] Key type .llcrypt unregistered [ 3593.575712] Key type ._llcrypt unregistered [ 3600.556879] Key type ._llcrypt registered [ 3600.557821] Key type .llcrypt registered [ 3600.592275] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 3600.952878] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3600.958896] alg: No test for adler32 (adler32-zlib) [ 3601.793186] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 3601.860790] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 3603.431110] Key type lgssc registered [ 3603.732735] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3606.081518] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3606.087073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3607.217564] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3608.358544] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3610.925643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3611.032314] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3612.191275] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3613.967742] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3615.205070] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3617.457835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3617.573042] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3619.309782] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3620.864831] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3631.912680] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 3638.776155] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3638.801104] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3638.891706] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3640.836926] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3642.657091] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3643.222309] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3653.899685] Lustre: Permanently deactivating lustre-OST0001 [ 3653.904942] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3655.034957] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 3664.647707] Lustre: Permanently reactivating lustre-OST0001 [ 3664.652934] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 3664.655171] Lustre: Skipped 2 previous similar messages [ 3673.081906] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3673.086494] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3673.089989] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3673.093048] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3673.104704] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3678.176655] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3678.177099] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3678.178722] LustreError: Skipped 1 previous similar message [ 3678.182307] Lustre: Skipped 1 previous similar message [ 3678.182804] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3678.186601] Lustre: Skipped 1 previous similar message [ 3679.867096] Lustre: server umount lustre-OST0001 complete [ 3683.296209] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3683.298382] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3683.301467] Lustre: Skipped 1 previous similar message [ 3683.303425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3683.306428] Lustre: Skipped 1 previous similar message [ 3683.809851] LustreError: 205152:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3683.817416] LustreError: 205152:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3687.184750] Lustre: server umount lustre-OST0000 complete [ 3688.928454] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3688.929386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3688.934587] Lustre: Skipped 2 previous similar messages [ 3688.937447] Lustre: Skipped 1 previous similar message [ 3694.048592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3694.052435] Lustre: Skipped 1 previous similar message [ 3694.557215] Lustre: server umount lustre-MDT0000 complete [ 3695.720831] LustreError: 203541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775887729 with bad export cookie 1856767055948184087 [ 3695.723517] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3695.724636] LustreError: 203541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3695.822985] Lustre: server umount lustre-MDT0001 complete [ 3699.066328] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 3700.034431] Key type lgssc unregistered [ 3700.175331] LNet: 208154:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3700.177559] LNetError: 208154:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3700.186388] LNet: Removed LNI 192.168.202.131@tcp [ 3700.444920] Key type .llcrypt unregistered [ 3700.446144] Key type ._llcrypt unregistered [ 3714.832688] Key type ._llcrypt registered [ 3714.834201] Key type .llcrypt registered [ 3714.870547] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 3715.434856] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3715.454879] alg: No test for adler32 (adler32-zlib) [ 3716.305693] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 3716.393576] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 3717.975148] Key type lgssc registered [ 3718.393202] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3720.972220] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3720.978192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3722.072117] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3722.080556] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3722.141703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3723.434962] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3726.297189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3726.329047] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3726.341402] Lustre: 210317:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3726.441660] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3727.826354] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3729.689077] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3730.981554] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3733.378696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3733.459464] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3733.540446] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3735.509725] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3736.278838] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 3737.352588] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3738.915722] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 3746.451910] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 3751.966167] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 3760.549747] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3770.180684] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 3789.767705] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 3789.770053] Lustre: Skipped 2 previous similar messages [ 3828.557570] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 3828.559699] Lustre: Skipped 4 previous similar messages [ 3901.538995] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 3901.541507] Lustre: Skipped 8 previous similar messages [ 3905.506296] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3905.507608] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3905.508618] LustreError: Skipped 1 previous similar message [ 3905.509119] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3905.515983] Lustre: Skipped 1 previous similar message [ 3909.135409] Lustre: server umount lustre-OST0000 complete [ 3910.624835] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3910.624996] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3910.625492] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3910.625498] Lustre: Skipped 1 previous similar message [ 3910.639636] Lustre: Skipped 1 previous similar message [ 3915.744160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3915.746292] Lustre: Skipped 2 previous similar messages [ 3916.569577] Lustre: server umount lustre-MDT0000 complete [ 3917.759230] LustreError: 212600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775887951 with bad export cookie 5624399961275439162 [ 3917.762321] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3917.763265] LustreError: 212600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3917.877132] Lustre: server umount lustre-MDT0001 complete [ 3921.494353] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 3922.488186] Key type lgssc unregistered [ 3922.619343] LNet: 214774:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3922.622604] LNetError: 214774:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3922.632294] LNet: Removed LNI 192.168.202.131@tcp [ 3922.918954] Key type .llcrypt unregistered [ 3922.920061] Key type ._llcrypt unregistered [ 3931.859854] Key type ._llcrypt registered [ 3931.861034] Key type .llcrypt registered [ 3931.896577] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 3932.293882] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3932.344294] alg: No test for adler32 (adler32-zlib) [ 3933.199870] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 3933.280082] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 3934.855205] Key type lgssc registered [ 3935.223299] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3937.790314] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3937.796231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3938.954672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3940.195027] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3942.958360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3943.082174] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3944.363373] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3946.130057] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3947.357958] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3949.570000] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3949.693801] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3951.496334] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug -1 all [ 3953.207080] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3964.075763] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 3973.600075] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3973.602801] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3973.606526] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3974.625684] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3974.625765] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3974.625831] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3974.628468] Lustre: Skipped 1 previous similar message [ 3978.453100] Lustre: server umount lustre-OST0000 complete [ 3979.744536] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3979.749369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3984.351974] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3984.354235] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3984.359148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3989.984222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3989.986510] Lustre: Skipped 2 previous similar messages [ 3994.079131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3994.141724] Lustre: server umount lustre-MDT0000 complete [ 3995.104830] LustreError: 216145:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3995.113786] LustreError: 216145:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3995.413036] LustreError: 217761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775888029 with bad export cookie 2805409726747052726 [ 3995.415921] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3995.416428] LustreError: 217761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3995.537270] Lustre: server umount lustre-MDT0001 complete [ 3999.162567] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 4000.175482] Key type lgssc unregistered [ 4000.304389] LNet: 219502:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4000.306618] LNetError: 219502:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4000.313268] LNet: Removed LNI 192.168.202.131@tcp [ 4000.602491] Key type .llcrypt unregistered [ 4000.603520] Key type ._llcrypt unregistered [ 4009.155137] Key type ._llcrypt registered [ 4009.156437] Key type .llcrypt registered [ 4009.195803] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 4009.537204] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4009.566501] alg: No test for adler32 (adler32-zlib) [ 4010.421261] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 4010.503249] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 4012.087192] Key type lgssc registered [ 4012.446314] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4275.430711] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 4276.516835] Key type lgssc unregistered [ 4276.648298] LNet: 221466:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4276.650490] LNetError: 221466:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4276.662268] LNet: Removed LNI 192.168.202.131@tcp [ 4276.921654] Key type .llcrypt unregistered [ 4276.922888] Key type ._llcrypt unregistered [ 4284.999251] Key type ._llcrypt registered [ 4285.000367] Key type .llcrypt registered [ 4285.042785] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_hostid [ 4291.431501] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 4291.877624] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4291.979175] alg: No test for adler32 (adler32-zlib) [ 4292.826944] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 4292.911208] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 4294.503176] Key type lgssc registered [ 4294.923587] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4305.313087] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 4080 sec ========== 02:18:59 (1775888339) [ 4305.961430] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 02:18:59 (1775888339) === [ 4307.052392] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 02:19:01 (1775888341) === [ 4319.996797] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 4320.922901] Key type lgssc unregistered [ 4321.039490] LNet: 226619:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4321.041733] LNetError: 226619:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4321.053268] LNet: Removed LNI 192.168.202.131@tcp [ 4321.336468] Key type .llcrypt unregistered [ 4321.337882] Key type ._llcrypt unregistered