[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 527282082 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002227] x2apic enabled [ 0.003005] Switched APIC routing to physical x2apic. [ 0.004007] kvm-guest: setup PV IPIs [ 0.006247] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008007] pid_max: default: 32768 minimum: 301 [ 0.009147] LSM: Security Framework initializing [ 0.010039] Yama: becoming mindful. [ 0.011028] SELinux: Initializing. [ 0.012073] *** VALIDATE selinux *** [ 0.020686] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024806] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025170] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026102] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027133] *** VALIDATE tmpfs *** [ 0.028492] *** VALIDATE proc *** [ 0.029251] *** VALIDATE cgroup *** [ 0.030012] *** VALIDATE cgroup2 *** [ 0.032044] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033157] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035032] Spectre V2 : User space: Vulnerable [ 0.036010] Speculative Store Bypass: Vulnerable [ 0.038857] debug: unmapping init [mem 0xffffffffa2e59000-0xffffffffa2e60fff] [ 0.040178] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.041716] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.042023] ... version: 2 [ 0.043010] ... bit width: 48 [ 0.044010] ... generic registers: 4 [ 0.045012] ... value mask: 0000ffffffffffff [ 0.046011] ... max period: 00007fffffffffff [ 0.047010] ... fixed-purpose events: 3 [ 0.048011] ... event mask: 000000070000000f [ 0.049243] rcu: Hierarchical SRCU implementation. [ 0.051145] smp: Bringing up secondary CPUs ... [ 0.052493] x86: Booting SMP configuration: [ 0.053023] .... node #0, CPUs: #1 #2 #3 [ 0.056325] smp: Brought up 1 node, 4 CPUs [ 0.057996] smpboot: Max logical packages: 1 [ 0.059011] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.217646] node 0 deferred pages initialised in 156ms [ 0.220312] devtmpfs: initialized [ 0.221408] x86/mm: Memory block size: 128MB [ 0.223799] gcov: version magic: 0x41383552 [ 0.227172] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.231092] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.234411] pinctrl core: initialized pinctrl subsystem [ 0.237206] [ 0.237880] ************************************************************* [ 0.241015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.244011] ** ** [ 0.246011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.249082] ** ** [ 0.252014] ** This means that this kernel is built to expose internal ** [ 0.254008] ** IOMMU data structures, which may compromise security on ** [ 0.256010] ** your system. ** [ 0.257008] ** ** [ 0.260011] ** If you see this message and you are not debugging the ** [ 0.261008] ** kernel, report this immediately to your vendor! ** [ 0.263013] ** ** [ 0.266011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.268008] ************************************************************* [ 0.270663] NET: Registered protocol family 16 [ 0.272438] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.275073] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.278070] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.282102] cpuidle: using governor menu [ 0.285031] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.288470] PCI: Using configuration type 1 for base access [ 0.290155] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.301045] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.303079] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.306177] cryptd: max_cpu_qlen set to 1000 [ 0.310376] ACPI: Added _OSI(Module Device) [ 0.312034] ACPI: Added _OSI(Processor Device) [ 0.315027] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.317024] ACPI: Added _OSI(Processor Aggregator Device) [ 0.323081] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.332398] ACPI: Interpreter enabled [ 0.334093] ACPI: PM: (supports S0 S3 S4 S5) [ 0.336024] ACPI: Using IOAPIC for interrupt routing [ 0.338152] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.341640] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.351955] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.354065] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.357030] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.361112] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.365758] acpiphp: Slot [2] registered [ 0.367169] acpiphp: Slot [5] registered [ 0.369086] acpiphp: Slot [6] registered [ 0.370103] acpiphp: Slot [7] registered [ 0.372139] acpiphp: Slot [8] registered [ 0.373104] acpiphp: Slot [9] registered [ 0.375105] acpiphp: Slot [10] registered [ 0.376204] acpiphp: Slot [3] registered [ 0.377125] acpiphp: Slot [4] registered [ 0.378085] acpiphp: Slot [11] registered [ 0.380128] acpiphp: Slot [12] registered [ 0.381130] acpiphp: Slot [13] registered [ 0.382080] acpiphp: Slot [14] registered [ 0.384065] acpiphp: Slot [15] registered [ 0.385073] acpiphp: Slot [16] registered [ 0.386055] acpiphp: Slot [17] registered [ 0.387179] acpiphp: Slot [18] registered [ 0.389111] acpiphp: Slot [19] registered [ 0.391109] acpiphp: Slot [20] registered [ 0.393302] acpiphp: Slot [21] registered [ 0.395170] acpiphp: Slot [22] registered [ 0.397100] acpiphp: Slot [23] registered [ 0.399076] acpiphp: Slot [24] registered [ 0.400140] acpiphp: Slot [25] registered [ 0.402143] acpiphp: Slot [26] registered [ 0.403150] acpiphp: Slot [27] registered [ 0.405174] acpiphp: Slot [28] registered [ 0.407113] acpiphp: Slot [29] registered [ 0.408149] acpiphp: Slot [30] registered [ 0.410150] acpiphp: Slot [31] registered [ 0.411088] PCI host bridge to bus 0000:00 [ 0.413033] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.415036] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.418037] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.420038] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.423040] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.426043] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.428221] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.431730] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.435406] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.446712] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.451527] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.454017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.456013] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.458016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.461654] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.464803] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.468043] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.470804] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.476022] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.489988] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.495025] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.501333] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.507024] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.514021] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.526021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.534828] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.545018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.551015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.569017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.579000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.587025] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.598028] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.626025] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.634726] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.640016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.647016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.666014] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.677789] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.692015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.700022] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.714041] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.728839] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.732015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.736010] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.750019] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.767925] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.771606] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.774453] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.778636] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.782322] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.790043] iommu: Default domain type: Passthrough [ 0.792586] SCSI subsystem initialized [ 0.793148] ACPI: bus type USB registered [ 0.795100] usbcore: registered new interface driver usbfs [ 0.796063] usbcore: registered new interface driver hub [ 0.798071] usbcore: registered new device driver usb [ 0.800151] pps_core: LinuxPPS API ver. 1 registered [ 0.801008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.804076] PTP clock support registered [ 0.806043] EDAC MC: Ver: 3.0.0 [ 0.807522] PCI: Using ACPI for IRQ routing [ 0.810867] NetLabel: Initializing [ 0.812016] NetLabel: domain hash size = 128 [ 0.813009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.815148] NetLabel: unlabeled traffic allowed by default [ 0.818144] vgaarb: loaded [ 0.819280] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.822013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.828014] clocksource: Switched to clocksource kvm-clock [ 0.942285] VFS: Disk quotas dquot_6.6.0 [ 0.943544] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.945817] *** VALIDATE ramfs *** [ 0.947060] *** VALIDATE hugetlbfs *** [ 0.948267] pnp: PnP ACPI init [ 0.950379] pnp: PnP ACPI: found 6 devices [ 0.966141] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.969362] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.971699] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.973839] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.976156] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.978758] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.981650] NET: Registered protocol family 2 [ 0.984120] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.988608] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.992430] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.997947] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.001406] TCP: Hash tables configured (established 65536 bind 65536) [ 1.004411] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.007659] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.009882] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.012461] NET: Registered protocol family 1 [ 1.015286] RPC: Registered named UNIX socket transport module. [ 1.017456] RPC: Registered udp transport module. [ 1.019131] RPC: Registered tcp transport module. [ 1.020785] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.022904] NET: Registered protocol family 44 [ 1.024784] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.027103] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.029927] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.033111] PCI: CLS 0 bytes, default 64 [ 1.036453] Unpacking initramfs... [ 2.471750] debug: unmapping init [mem 0xffff94353cc54000-0xffff94353ffbffff] [ 2.478519] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.480533] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.484139] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.976845] Initialise system trusted keyrings [ 2.978172] Key type blacklist registered [ 2.980165] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.988916] zbud: loaded [ 2.992100] *** VALIDATE nfs *** [ 2.993361] *** VALIDATE nfs4 *** [ 2.994572] pstore: using deflate compression [ 2.997855] Platform Keyring initialized [ 3.098624] NET: Registered protocol family 38 [ 3.100019] Key type asymmetric registered [ 3.101107] Asymmetric key parser 'x509' registered [ 3.102471] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.108318] io scheduler mq-deadline registered [ 3.110054] io scheduler kyber registered [ 3.112068] io scheduler bfq registered [ 3.114105] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.119685] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.123239] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.128114] ACPI: Power Button [PWRF] [ 3.132551] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.137910] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.149289] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.154321] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.165402] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.193771] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.222398] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.227725] Non-volatile memory driver v1.3 [ 3.228863] Linux agpgart interface v0.103 [ 3.262457] virtio_blk virtio1: [vda] 149408 512-byte logical blocks (76.5 MB/73.0 MiB) [ 3.265164] vda: detected capacity change from 0 to 76496896 [ 3.282874] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.285931] vdb: detected capacity change from 0 to 1073741824 [ 3.302673] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.305249] vdc: detected capacity change from 0 to 2621440000 [ 3.321560] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.323984] vdd: detected capacity change from 0 to 2621440000 [ 3.339594] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.342738] vde: detected capacity change from 0 to 4294967296 [ 3.363652] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.376583] vdf: detected capacity change from 0 to 4294967296 [ 3.397655] libphy: Fixed MDIO Bus: probed [ 3.404455] usbcore: registered new interface driver usbserial_generic [ 3.408221] usbserial: USB Serial support registered for generic [ 3.411621] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.415575] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.417082] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.419723] mousedev: PS/2 mouse device common for all mice [ 3.424185] rtc_cmos 00:05: RTC can wake from S4 [ 3.427124] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.428671] rtc_cmos 00:05: registered as rtc0 [ 3.431273] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.436538] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.437499] intel_pstate: CPU model not supported [ 3.443310] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.449248] hid: raw HID events driver (C) Jiri Kosina [ 3.451760] usbcore: registered new interface driver usbhid [ 3.453977] usbhid: USB HID core driver [ 3.455844] drop_monitor: Initializing network drop monitor service [ 3.458440] Initializing XFRM netlink socket [ 3.460869] NET: Registered protocol family 10 [ 3.464422] Segment Routing with IPv6 [ 3.465413] NET: Registered protocol family 17 [ 3.467441] mpls_gso: MPLS GSO support [ 3.474095] RAS: Correctable Errors collector initialized. [ 3.476180] AVX version of gcm_enc/dec engaged. [ 3.477847] AES CTR mode by8 optimization enabled [ 3.558985] sched_clock: Marking stable (3558953575, 0)->(4438889326, -879935751) [ 3.563243] registered taskstats version 1 [ 3.567121] Loading compiled-in X.509 certificates [ 3.569327] zswap: loaded using pool lzo/zbud [ 3.596665] Key type big_key registered [ 3.609658] Key type encrypted registered [ 3.611488] ima: No TPM chip found, activating TPM-bypass! [ 3.613810] ima: Allocated hash algorithm: sha1 [ 3.615268] ima: No architecture policies found [ 3.617560] evm: Initialising EVM extended attributes: [ 3.619616] evm: security.selinux [ 3.620869] evm: security.ima [ 3.622204] evm: security.capability [ 3.623590] evm: HMAC attrs: 0x1 [ 3.625592] rtc_cmos 00:05: setting system clock to 2026-08-22 09:42:17 UTC (1787391737) [ 3.630308] debug: unmapping init [mem 0xffffffffa3e03000-0xffffffffa3ffffff] [ 3.632482] debug: unmapping init [mem 0xffffffffa2b82000-0xffffffffa2e58fff] [ 3.643128] Write protecting the kernel read-only data: 28672k [ 3.646608] debug: unmapping init [mem 0xffffffffa1203000-0xffffffffa13fffff] [ 3.648891] debug: unmapping init [mem 0xffffffffa1b14000-0xffffffffa1bfffff] [ 3.691700] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.699845] systemd[1]: Detected virtualization kvm. [ 3.701534] systemd[1]: Detected architecture x86-64. [ 3.703815] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.731210] systemd[1]: No hostname configured. [ 3.733050] systemd[1]: Set hostname to . [ 3.735878] random: systemd: uninitialized urandom read (16 bytes read) [ 3.738485] systemd[1]: Initializing machine ID from random generator. [ 3.900783] random: systemd: uninitialized urandom read (16 bytes read) [ 3.902977] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.907519] random: systemd: uninitialized urandom read (16 bytes read) [ 3.910076] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.916102] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Reached target Swap. [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Sockets. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... Starting Journal Service... [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Reached target Initrd Root Device. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Slices. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.559069] device-mapper: uevent: version 1.0.3 [ 4.562296] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.386827] random: fast init done [ 5.414175] virtio_net virtio0 ens2: renamed from eth0 [ 5.443071] scsi host0: ata_piix [ 5.457685] scsi host1: ata_piix [ 5.459439] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.462083] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.976289] dracut-initqueue[587]: RTNETLINK answers: File exists [ 10.129745] random: crng init done [ 10.131065] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.695993] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.880604] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.142390] SELinux: Disabled at runtime. [ 12.203931] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.213611] systemd[1]: Detected virtualization kvm. [ 12.215662] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.724496] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.728585] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.734032] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.739437] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.743318] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.750347] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.757319] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Apply Kernel Variables... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Initrd Root File System. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target rpc_pipefs.target. Mounting Kernel Debug File System... Starting Remount Root and Kernel File Systems... [ 12.825484] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Huge Pages File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on udev Control Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting POSIX Message Queue File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.195071] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.393606] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.467425] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.579722] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.603245] EDAC sbridge: Ver: 1.1.2 [ 15.320827] Key type dns_resolver registered [ 15.602719] NFS: Registering the id_resolver key type [ 15.604522] Key type id_resolver registered [ 15.605464] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Started Login Service. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg205-server login: [ 56.844276] libcfs: loading out-of-tree module taints kernel. [ 56.895158] Key type ._llcrypt registered [ 56.897288] Key type .llcrypt registered [ 56.988771] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_hostid [ 80.669668] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 83.177270] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 83.221699] alg: No test for adler32 (adler32-zlib) [ 84.787943] Lustre: Lustre: Build Version: 2.17.54_110_g2f98931 [ 85.790838] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 87.536656] Key type lgssc registered [ 89.877600] Lustre: Echo OBD driver; http://www.lustre.org/ [ 107.691661] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 130.051014] hrtimer: interrupt took 9018427 ns [ 150.052728] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 163.234291] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 163.286087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 164.590945] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 164.616243] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 164.763748] Lustre: lustre-MDT0000: new disk, initializing [ 164.857485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 164.873860] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 169.395866] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 184.429033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 184.581398] Lustre: 6513:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 184.643698] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 184.654607] Lustre: Skipped 1 previous similar message [ 184.783626] Lustre: lustre-MDT0001: new disk, initializing [ 184.868272] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 184.917881] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 184.925202] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 190.585337] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 195.548988] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 206.668812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 207.023246] Lustre: lustre-OST0000: new disk, initializing [ 207.025522] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 207.030846] Lustre: 8451:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 207.119901] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 213.995000] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 216.633227] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 216.651517] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 216.716372] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 229.708823] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 229.801334] Lustre: lustre-OST0001: new disk, initializing [ 229.803353] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 229.806309] Lustre: 9523:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 229.850847] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 237.525905] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 239.734326] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 239.749785] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 239.791686] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 251.505322] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 259.831957] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 267.816758] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing check_logdir /tmp/testlogs/ [ 274.170558] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing yml_node [ 278.937531] Lustre: DEBUG MARKER: Client: 2.17.54.110 [ 282.019818] Lustre: DEBUG MARKER: MDS: 2.17.54.110 [ 285.129571] Lustre: DEBUG MARKER: OSS: 2.17.54.110 [ 286.938883] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Aug 22 05:46:59 EDT 2026 [ 308.034768] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 309.719126] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 327.138270] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 327.140501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 327.150665] Lustre: Skipped 3 previous similar messages [ 327.155404] Lustre: Skipped 2 previous similar messages [ 332.301809] Lustre: server umount lustre-MDT0000 complete [ 335.957217] LustreError: 6504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392069 with bad export cookie 7937703204210834659 [ 335.960139] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 335.971902] LustreError: 6504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 337.376148] LustreError: 10149:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 337.376520] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 337.397881] LustreError: 10149:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 337.412512] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 337.416969] Lustre: Skipped 2 previous similar messages [ 340.384321] LustreError: 10149:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 340.384513] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 340.410898] Lustre: Skipped 1 previous similar message [ 342.611531] Lustre: server umount lustre-MDT0001 complete [ 346.311683] Lustre: server umount lustre-OST0000 complete [ 350.320528] Lustre: server umount lustre-OST0001 complete [ 357.208939] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_hostid [ 368.159775] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 410.368061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 410.702211] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 410.727894] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 410.789910] Lustre: lustre-MDT0000: new disk, initializing [ 410.853958] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 410.870539] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 415.312837] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 429.461440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 429.637758] Lustre: 15082:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 429.799270] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 429.806109] Lustre: Skipped 1 previous similar message [ 429.982490] Lustre: lustre-MDT0001: new disk, initializing [ 430.144195] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 430.214963] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 430.245273] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 435.619927] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 445.517244] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 452.276918] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 459.773561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 460.038985] Lustre: lustre-OST0000: new disk, initializing [ 460.042850] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 460.048809] Lustre: 17051:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 460.159559] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 462.141440] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 462.156648] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 462.285657] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 466.793823] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 477.901918] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 484.078650] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 484.292386] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 488.685210] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 488.859667] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 493.040608] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 493.049091] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 493.060280] Lustre: Skipped 1 previous similar message [ 493.071422] LustreError: Skipped 1 previous similar message [ 493.073775] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 493.079388] Lustre: Skipped 1 previous similar message [ 497.599603] Lustre: server umount lustre-OST0000 complete [ 501.728805] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 501.736806] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 501.748150] Lustre: Skipped 1 previous similar message [ 501.760686] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 507.920824] Lustre: server umount lustre-MDT0000 complete [ 508.384918] LustreError: 15094:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 508.405486] LustreError: 15094:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 512.577875] LustreError: 15073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392246 with bad export cookie 7937703204210836192 [ 512.580964] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 512.587023] LustreError: 15073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 512.995921] Lustre: server umount lustre-MDT0001 complete [ 516.827532] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 05:50:48 (1787392248) [ 523.549839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 524.091857] LustreError: 19317:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 524.168405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 528.567396] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 529.388771] LustreError: 19318:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 536.342471] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 541.074045] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 548.487743] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 556.690704] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 564.141424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 571.969294] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 579.776860] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 591.848593] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 591.858070] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 591.877288] Lustre: Skipped 1 previous similar message [ 591.886776] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 591.892348] Lustre: Skipped 2 previous similar messages [ 596.341313] Lustre: server umount lustre-OST0000 complete [ 602.593541] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 602.615981] Lustre: Skipped 2 previous similar messages [ 608.643775] Lustre: server umount lustre-MDT0000 complete [ 612.832662] LustreError: 19317:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 612.872743] LustreError: 19317:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 614.325874] LustreError: 19297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392348 with bad export cookie 7937703204210837074 [ 614.346665] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 614.655110] Lustre: server umount lustre-MDT0001 complete [ 627.823979] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 632.513327] Key type lgssc unregistered [ 632.912084] LNet: 22850:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 632.934621] LNetError: 22850:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 632.966155] LNet: Removed LNI 192.168.202.105@tcp [ 634.321194] Key type .llcrypt unregistered [ 634.325243] Key type ._llcrypt unregistered [ 655.273870] Key type ._llcrypt registered [ 655.277186] Key type .llcrypt registered [ 655.396609] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 656.874432] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 656.998731] alg: No test for adler32 (adler32-zlib) [ 658.241266] Lustre: Lustre: Build Version: 2.17.54_110_g2f98931 [ 658.500727] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 660.215237] Key type lgssc registered [ 661.627374] Lustre: Echo OBD driver; http://www.lustre.org/ [ 670.275254] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 670.309659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 671.904460] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 678.041478] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 681.446917] LustreError: 24244:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 681.471032] LustreError: 24244:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 686.595701] LustreError: 24243:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 688.748868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 689.178476] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 696.833980] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 705.523905] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 711.253657] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 718.366418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 718.925735] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 725.766699] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 733.589291] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 741.498660] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 752.107443] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 752.112937] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 752.118263] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 752.130657] Lustre: Skipped 1 previous similar message [ 757.230395] LustreError: 26067:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 757.254630] LustreError: 26067:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 757.365818] Lustre: server umount lustre-OST0000 complete [ 760.800631] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 760.810977] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 760.833222] Lustre: Skipped 1 previous similar message [ 760.838396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 762.337562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 762.345421] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 766.971891] Lustre: server umount lustre-MDT0000 complete [ 767.456720] LustreError: 24238:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 767.485424] LustreError: 24238:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 771.287289] LustreError: 24224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392505 with bad export cookie 16666825456175551652 [ 771.299917] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 771.299926] LustreError: 24224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 771.645682] Lustre: server umount lustre-MDT0001 complete [ 783.519506] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 786.362791] Key type lgssc unregistered [ 786.615267] LNet: 28222:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 786.625052] LNetError: 28222:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 786.650558] LNet: Removed LNI 192.168.202.105@tcp [ 787.698324] Key type .llcrypt unregistered [ 787.705138] Key type ._llcrypt unregistered [ 815.194711] Key type ._llcrypt registered [ 815.208340] Key type .llcrypt registered [ 815.326585] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 816.346988] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 816.451729] alg: No test for adler32 (adler32-zlib) [ 817.842753] Lustre: Lustre: Build Version: 2.17.54_110_g2f98931 [ 818.151054] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 819.871468] Key type lgssc registered [ 821.424539] Lustre: Echo OBD driver; http://www.lustre.org/ [ 830.102621] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 830.164213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 831.671239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 835.957210] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 841.251236] LustreError: 29637:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 841.265762] LustreError: 29637:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 845.492828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 845.921870] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 850.998447] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 858.494740] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 864.289768] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 877.902976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 878.127689] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 883.882902] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 891.263089] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 901.686557] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 910.818413] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 910.828113] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 910.847243] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 911.844299] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 911.844670] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 911.869398] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 913.974596] Lustre: server umount lustre-OST0000 complete [ 922.089326] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 922.128731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 922.139382] Lustre: Skipped 1 previous similar message [ 925.466802] Lustre: server umount lustre-MDT0000 complete [ 927.206734] LustreError: 32472:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 927.242912] LustreError: 32472:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 930.859889] LustreError: 29619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392664 with bad export cookie 3773350987238870187 [ 930.885696] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 931.326732] Lustre: server umount lustre-MDT0001 complete [ 946.861280] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 950.709466] Key type lgssc unregistered [ 951.074833] LNet: 33540:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 951.098705] LNetError: 33540:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 951.127348] LNet: Removed LNI 192.168.202.105@tcp [ 952.344672] Key type .llcrypt unregistered [ 952.346841] Key type ._llcrypt unregistered [ 980.364899] Key type ._llcrypt registered [ 980.366561] Key type .llcrypt registered [ 980.525697] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 981.534975] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 981.561221] alg: No test for adler32 (adler32-zlib) [ 982.975715] Lustre: Lustre: Build Version: 2.17.54_110_g2f98931 [ 983.193893] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 984.855421] Key type lgssc registered [ 986.294272] Lustre: Echo OBD driver; http://www.lustre.org/ [ 995.665178] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 995.694953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 997.094286] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1003.533588] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 1006.564634] LustreError: 34956:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1006.582222] LustreError: 34956:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1011.690899] LustreError: 34955:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1014.020348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1014.635857] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1020.302872] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 1030.568868] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1036.758207] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1045.623338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1046.201121] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1055.465756] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 1065.118522] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1066.685357] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1082.337510] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1082.351735] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1082.356258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1082.370557] Lustre: Skipped 1 previous similar message [ 1086.734899] Lustre: server umount lustre-OST0000 complete [ 1092.577610] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1092.590728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1092.595712] Lustre: Skipped 1 previous similar message [ 1092.620559] Lustre: Skipped 2 previous similar messages [ 1097.697335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1097.705500] Lustre: Skipped 1 previous similar message [ 1098.188163] Lustre: server umount lustre-MDT0000 complete [ 1102.818852] LustreError: 37448:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1102.858436] LustreError: 37448:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1103.405528] LustreError: 34936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392837 with bad export cookie 17856277516266499872 [ 1103.415623] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1103.420462] LustreError: 34936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1104.011239] Lustre: server umount lustre-MDT0001 complete [ 1117.191310] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 1120.818447] Key type lgssc unregistered [ 1121.160264] LNet: 38509:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1121.167451] LNetError: 38509:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1121.184682] LNet: Removed LNI 192.168.202.105@tcp [ 1122.059358] Key type .llcrypt unregistered [ 1122.063874] Key type ._llcrypt unregistered [ 1151.156813] Key type ._llcrypt registered [ 1151.164553] Key type .llcrypt registered [ 1151.294667] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 1152.524736] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1152.643055] alg: No test for adler32 (adler32-zlib) [ 1153.884666] Lustre: Lustre: Build Version: 2.17.54_110_g2f98931 [ 1154.123035] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 1155.841180] Key type lgssc registered [ 1156.901495] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1164.188538] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1164.211905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1165.742761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1170.711403] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 1176.037501] LustreError: 39937:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1176.052106] LustreError: 39937:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1181.072469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1181.160501] LustreError: 39936:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1181.437772] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1187.484637] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 1195.827736] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1202.504736] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1209.644640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1210.047872] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1211.136037] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1217.647935] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 1225.854747] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1232.820609] Lustre: lustre-OST0000: Not available for connect from 192.168.202.5@tcp (stopping) [ 1236.960314] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1236.961586] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1236.963861] LustreError: Skipped 1 previous similar message [ 1236.977155] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1236.995671] Lustre: Skipped 1 previous similar message [ 1237.036224] Lustre: Skipped 1 previous similar message [ 1238.659974] Lustre: server umount lustre-OST0000 complete [ 1261.537722] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1261.542660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1261.564228] Lustre: Skipped 1 previous similar message [ 1261.582014] Lustre: Skipped 2 previous similar messages [ 1266.656935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1271.780710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1271.800581] Lustre: Skipped 2 previous similar messages [ 1275.359141] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1275.647741] Lustre: server umount lustre-MDT0000 complete [ 1276.896755] LustreError: 40876:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1276.944223] LustreError: 40876:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1279.994784] LustreError: 41781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393013 with bad export cookie 9655991997967240537 [ 1280.007120] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1280.010409] LustreError: 41781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1286.809058] Lustre: server umount lustre-MDT0001 complete [ 1300.463767] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 1303.658329] Key type lgssc unregistered [ 1303.941687] LNet: 43648:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1303.961158] LNetError: 43648:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1305.010155] LNet: Removed LNI 192.168.202.105@tcp [ 1306.064257] Key type .llcrypt unregistered [ 1306.067747] Key type ._llcrypt unregistered [ 1332.889675] Key type ._llcrypt registered [ 1332.893936] Key type .llcrypt registered [ 1333.021418] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 1334.061293] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1334.244629] alg: No test for adler32 (adler32-zlib) [ 1335.292261] Lustre: Lustre: Build Version: 2.17.54_110_g2f98931 [ 1335.588540] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 1337.343209] Key type lgssc registered [ 1339.064724] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1348.266198] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1348.293501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1349.919288] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1355.780691] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 1359.334987] LustreError: 45062:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1359.374828] LustreError: 45062:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1364.458581] LustreError: 45063:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1366.530557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1366.949347] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1373.005639] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 1384.130608] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1392.991467] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1402.401134] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1403.343026] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1410.792879] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 1419.803320] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1420.172533] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1428.414996] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.5@tcp (stopping) [ 1428.448857] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1428.459951] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1429.984246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1429.985187] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1429.992651] Lustre: Skipped 1 previous similar message [ 1430.011642] Lustre: Skipped 1 previous similar message [ 1433.494194] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.5@tcp (stopping) [ 1433.504922] Lustre: Skipped 2 previous similar messages [ 1438.609173] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.5@tcp (stopping) [ 1438.618680] Lustre: Skipped 3 previous similar messages [ 1442.271920] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1442.453929] Lustre: server umount lustre-MDT0000 complete [ 1443.733434] LustreError: 46159:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1446.641608] LustreError: 45043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393180 with bad export cookie 8744904406311189765 [ 1446.648593] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1446.655900] LustreError: 45043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1448.853737] LustreError: 45057:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1448.878272] LustreError: 45057:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1448.897379] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.5@tcp (stopping) [ 1448.901466] Lustre: Skipped 3 previous similar messages [ 1450.466121] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1453.166933] Lustre: server umount lustre-MDT0001 complete [ 1469.305272] Lustre: server umount lustre-OST0000 complete [ 1492.138727] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 1495.087781] Key type lgssc unregistered [ 1495.388777] LNet: 48926:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1495.395781] LNetError: 48926:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1495.418235] LNet: Removed LNI 192.168.202.105@tcp [ 1496.364730] Key type .llcrypt unregistered [ 1496.366399] Key type ._llcrypt unregistered [ 1529.727499] Key type ._llcrypt registered [ 1529.729873] Key type .llcrypt registered [ 1529.817441] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 1531.198485] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1531.360141] alg: No test for adler32 (adler32-zlib) [ 1532.444496] Lustre: Lustre: Build Version: 2.17.54_110_g2f98931 [ 1532.652078] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 1534.327195] Key type lgssc registered [ 1535.744355] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1545.087783] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1545.104546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1560.609707] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1565.727432] LustreError: 50319:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.105@tcp: failed processing log, type 4: rc = -110 [ 1592.416211] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1599.435395] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 1607.886533] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1774.407475] Lustre: server umount lustre-OST0000 complete [ 1795.256821] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 1798.187842] Key type lgssc unregistered [ 1798.540848] LNet: 51996:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1798.548116] LNetError: 51996:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1798.578387] LNet: Removed LNI 192.168.202.105@tcp [ 1799.281082] Key type .llcrypt unregistered [ 1799.289943] Key type ._llcrypt unregistered [ 1832.734855] Key type ._llcrypt registered [ 1832.737255] Key type .llcrypt registered [ 1832.855285] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 1834.176425] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1834.248869] alg: No test for adler32 (adler32-zlib) [ 1835.538962] Lustre: Lustre: Build Version: 2.17.54_110_g2f98931 [ 1835.964835] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 1837.600670] Key type lgssc registered [ 1839.233307] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1848.404891] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1848.421546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1849.881726] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1854.810683] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 1860.065392] LustreError: 53414:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1860.099843] LustreError: 53414:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1863.964142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1864.504271] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1870.123757] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 1879.286643] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1886.166698] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1894.262484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1894.433494] LustreError: 55234:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1894.823322] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1900.018691] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1901.933750] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 1909.837385] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1914.339427] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1914.350750] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1914.375141] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1914.389528] Lustre: Skipped 1 previous similar message [ 1919.171882] Lustre: server umount lustre-OST0000 complete [ 1924.582356] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1924.591300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1924.610065] Lustre: Skipped 2 previous similar messages [ 1924.632979] Lustre: Skipped 1 previous similar message [ 1929.697669] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1929.714419] Lustre: Skipped 1 previous similar message [ 1934.816419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1934.822263] Lustre: Skipped 1 previous similar message [ 1938.402518] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1938.690611] Lustre: server umount lustre-MDT0000 complete [ 1939.936933] LustreError: 53415:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1939.978252] LustreError: 53415:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1943.767045] LustreError: 53394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393677 with bad export cookie 13349954381083882420 [ 1943.775734] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1943.782977] LustreError: 53394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1944.390727] Lustre: server umount lustre-MDT0001 complete [ 1960.373549] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 1964.232823] Key type lgssc unregistered [ 1964.735678] LNet: 56950:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1964.739157] LNetError: 56950:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1964.755201] LNet: Removed LNI 192.168.202.105@tcp [ 1966.011196] Key type .llcrypt unregistered [ 1966.018155] Key type ._llcrypt unregistered [ 1992.529953] Key type ._llcrypt registered [ 1992.533081] Key type .llcrypt registered [ 1992.668893] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 1994.216189] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1994.240285] alg: No test for adler32 (adler32-zlib) [ 1995.270793] Lustre: Lustre: Build Version: 2.17.54_110_g2f98931 [ 1995.484856] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 1997.159348] Key type lgssc registered [ 1998.408731] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2006.237312] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2006.252234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2021.855745] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2026.975318] LustreError: 58336:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.105@tcp: failed processing log, type 4: rc = -110 [ 2053.727303] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2062.825961] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2071.806936] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2080.956996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2081.582086] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2081.690284] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2086.894419] LustreError: 58362:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2086.934187] LustreError: 58362:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2087.707233] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2088.932427] LustreError: 58361:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2088.953810] LustreError: 58361:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2094.072588] LustreError: 59370:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2094.098947] LustreError: 59370:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2097.802249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2098.074992] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2103.050806] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2110.475372] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2116.285323] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2118.639712] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2118.645138] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2118.655547] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2118.673586] Lustre: Skipped 1 previous similar message [ 2118.701890] Lustre: Skipped 1 previous similar message [ 2123.755269] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2123.764846] Lustre: Skipped 1 previous similar message [ 2124.564575] Lustre: server umount lustre-OST0000 complete [ 2148.388325] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2148.398544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2148.408029] Lustre: Skipped 1 previous similar message [ 2148.424030] Lustre: Skipped 1 previous similar message [ 2152.691128] Lustre: server umount lustre-MDT0000 complete [ 2153.440359] LustreError: 61067:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2153.461500] LustreError: 61067:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2156.581571] LustreError: 58344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393890 with bad export cookie 14860307101180584528 [ 2156.599146] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2156.600423] LustreError: 58344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2163.002695] Lustre: server umount lustre-MDT0001 complete [ 2175.195121] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 2178.458844] Key type lgssc unregistered [ 2178.840878] LNet: 62082:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2178.849764] LNetError: 62082:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2178.868411] LNet: Removed LNI 192.168.202.105@tcp [ 2180.081128] Key type .llcrypt unregistered [ 2180.083793] Key type ._llcrypt unregistered [ 2203.666178] Key type ._llcrypt registered [ 2203.671134] Key type .llcrypt registered [ 2203.780445] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 2205.106497] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2205.275397] alg: No test for adler32 (adler32-zlib) [ 2206.483327] Lustre: Lustre: Build Version: 2.17.54_110_g2f98931 [ 2206.712221] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 2208.352546] Key type lgssc registered [ 2209.816645] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2218.728885] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2218.762334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2220.350694] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2225.316376] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2230.754771] LustreError: 63503:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2230.782652] LustreError: 63503:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2234.732662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2235.240901] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2239.862330] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2247.426859] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2253.272480] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2260.371604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2260.673206] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2265.775684] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2266.961538] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2275.509119] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2295.784456] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2295.788454] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2295.792479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2295.794256] Lustre: Skipped 1 previous similar message [ 2295.808477] Lustre: Skipped 1 previous similar message [ 2300.899200] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2306.025633] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2306.030753] Lustre: Skipped 2 previous similar messages [ 2308.063540] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2308.511542] Lustre: server umount lustre-OST0000 complete [ 2316.272220] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2316.275896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2316.299525] Lustre: Skipped 1 previous similar message [ 2316.313882] Lustre: Skipped 1 previous similar message [ 2318.912424] Lustre: server umount lustre-MDT0000 complete [ 2321.378835] LustreError: 63503:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2321.405032] LustreError: 63503:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2322.900383] LustreError: 63483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394056 with bad export cookie 11599309569096876458 [ 2322.914022] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2323.211868] Lustre: server umount lustre-MDT0001 complete [ 2332.731454] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 2334.992262] Key type lgssc unregistered [ 2335.247969] LNet: 67059:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2335.254219] LNetError: 67059:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2335.269993] LNet: Removed LNI 192.168.202.105@tcp [ 2336.122141] Key type .llcrypt unregistered [ 2336.124630] Key type ._llcrypt unregistered [ 2361.512971] Key type ._llcrypt registered [ 2361.514463] Key type .llcrypt registered [ 2361.717323] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 2362.542388] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2362.750071] alg: No test for adler32 (adler32-zlib) [ 2363.721785] Lustre: Lustre: Build Version: 2.17.54_110_g2f98931 [ 2363.986451] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 2365.679216] Key type lgssc registered [ 2367.026422] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2376.834603] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 06:21:48 (1787394108) [ 2386.561159] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2386.599327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2388.480308] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2393.787403] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2398.697045] LustreError: 68762:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2398.733559] LustreError: 68762:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2402.468326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2402.840094] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2407.821330] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2415.367423] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2421.455728] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2427.550212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2427.991911] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2433.149466] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2434.416794] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2443.135510] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2452.562996] Lustre: Failing over lustre-MDT0000 [ 2452.806878] Lustre: server umount lustre-MDT0000 complete [ 2452.969752] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2452.984807] Lustre: Skipped 2 previous similar messages [ 2452.987860] LustreError: 68762:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2453.033115] LustreError: 68762:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2455.488630] LustreError: 68758:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2458.081219] LustreError: 69489:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2458.108296] LustreError: 69489:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2463.209907] LustreError: 70194:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2463.250511] LustreError: 70194:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2463.834614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2463.922194] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2464.023920] Lustre: *** cfs_fail_loc=135, val=0*** [ 2464.110084] LustreError: 71849:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2464.114390] LustreError: 71849:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.105@tcp: cfg command failed: rc = -2 [ 2464.121839] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2464.121839] [ 2464.130081] LustreError: MGC192.168.202.105@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2464.142738] LustreError: 71837:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2464.152494] LustreError: 71837:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2464.160836] LustreError: 71837:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2464.165033] LustreError: 71837:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2464.182480] Lustre: server umount lustre-MDT0000 complete [ 2464.184905] LustreError: 71837:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2470.922000] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2471.140873] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2471.423905] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2471.502473] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2471.879380] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2476.013356] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2476.527582] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2476.560762] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2476.628095] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2486.758636] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2486.767349] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2486.783127] Lustre: Skipped 1 previous similar message [ 2486.808177] Lustre: Skipped 1 previous similar message [ 2491.618791] Lustre: server umount lustre-OST0000 complete [ 2496.399419] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2496.440747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2496.447557] Lustre: Skipped 1 previous similar message [ 2501.601157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2501.607648] Lustre: Skipped 1 previous similar message [ 2502.686725] Lustre: server umount lustre-MDT0000 complete [ 2506.729795] LustreError: 68762:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2506.760729] LustreError: 68762:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 2506.913740] LustreError: 68744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394240 with bad export cookie 4269288257314391143 [ 2506.914761] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2506.924162] LustreError: 68744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2507.329989] Lustre: server umount lustre-MDT0001 complete [ 2520.172658] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 2523.183248] Key type lgssc unregistered [ 2523.528241] LNet: 73629:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2523.539369] LNetError: 73629:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2523.560683] LNet: Removed LNI 192.168.202.105@tcp [ 2524.966458] Key type .llcrypt unregistered [ 2524.968151] Key type ._llcrypt unregistered [ 2556.365225] Key type ._llcrypt registered [ 2556.367358] Key type .llcrypt registered [ 2556.475802] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 2558.146316] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2558.194338] alg: No test for adler32 (adler32-zlib) [ 2559.324520] Lustre: Lustre: Build Version: 2.17.54_110_g2f98931 [ 2559.766364] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 2561.551171] Key type lgssc registered [ 2562.949986] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2572.091798] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2572.122903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2573.738911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2578.909668] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2584.037413] LustreError: 75046:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2584.059534] LustreError: 75046:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2589.152295] LustreError: 75047:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2589.399193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2589.906658] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2595.041699] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2606.552411] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2614.977079] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2624.114411] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2624.739598] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2631.164353] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2639.266333] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2643.186513] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2651.093423] Lustre: Failing over lustre-MDT0000 [ 2651.595855] LustreError: 77166:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2651.618143] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2651.621661] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2651.762225] Lustre: server umount lustre-MDT0000 complete [ 2653.670678] LustreError: 75047:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2653.676254] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2653.697986] LustreError: 75047:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2658.794236] LustreError: 75046:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2658.824415] LustreError: 75046:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2662.279171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2662.480324] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2662.721401] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2662.947164] LustreError: 78137:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2662.970080] LustreError: 78137:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.105@tcp: cfg command failed: rc = -9 [ 2662.982598] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2662.982598] [ 2662.997535] LustreError: MGC192.168.202.105@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2663.027636] LustreError: 78125:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2663.046762] LustreError: 78125:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 2663.065225] LustreError: 78125:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2663.074949] LustreError: 78125:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2663.110161] Lustre: server umount lustre-MDT0000 complete [ 2663.112264] LustreError: 78125:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2666.912522] LustreError: 75041:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2666.932453] LustreError: 75041:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2669.734856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2669.916627] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2670.297965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2670.369050] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2673.047370] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2675.718120] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2675.770335] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 2675.837338] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2675.970910] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2685.927051] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2685.934319] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2685.938062] Lustre: Skipped 2 previous similar messages [ 2685.945830] Lustre: Skipped 1 previous similar message [ 2691.062811] LustreError: 76871:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2691.079962] LustreError: 76871:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2691.187505] Lustre: server umount lustre-OST0000 complete [ 2696.163713] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2696.171337] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2696.181686] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2696.218372] Lustre: Skipped 1 previous similar message [ 2701.289350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2701.293971] Lustre: Skipped 2 previous similar messages [ 2704.420956] Lustre: server umount lustre-MDT0000 complete [ 2708.481818] LustreError: 75026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394442 with bad export cookie 7067803917312050538 [ 2708.488281] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2708.493589] LustreError: 75026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2708.758789] Lustre: server umount lustre-MDT0001 complete [ 2720.217565] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 2723.005645] Key type lgssc unregistered [ 2723.301651] LNet: 79917:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2723.308684] LNetError: 79917:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2724.332200] LNet: Removed LNI 192.168.202.105@tcp [ 2725.232908] Key type .llcrypt unregistered [ 2725.235233] Key type ._llcrypt unregistered [ 2738.660990] Key type ._llcrypt registered [ 2738.663422] Key type .llcrypt registered [ 2738.780315] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 2739.091871] Key type .llcrypt unregistered [ 2739.096474] Key type ._llcrypt unregistered [ 2765.267888] Key type ._llcrypt registered [ 2765.269966] Key type .llcrypt registered [ 2765.393368] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 2766.507823] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2766.540245] alg: No test for adler32 (adler32-zlib) [ 2767.544439] Lustre: Lustre: Build Version: 2.17.54_110_g2f98931 [ 2767.744165] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 2769.439504] Key type lgssc registered [ 2770.671425] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2779.274447] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2779.306344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2780.932945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2786.268820] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2790.374111] LustreError: 81821:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2790.406241] LustreError: 81821:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2795.488457] LustreError: 81820:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2796.872714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2797.506409] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2802.358835] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2811.350624] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2816.295300] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2822.869066] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2823.140613] LustreError: 83649:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2823.430463] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2825.563186] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2832.185508] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2842.525066] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2856.419705] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2856.421543] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2856.434789] LustreError: Skipped 1 previous similar message [ 2856.456856] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2856.465284] Lustre: Skipped 1 previous similar message [ 2856.480433] Lustre: Skipped 1 previous similar message [ 2861.542690] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2861.557892] Lustre: Skipped 1 previous similar message [ 2861.946733] Lustre: server umount lustre-OST0000 complete [ 2866.661397] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2866.689654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2866.700633] Lustre: Skipped 1 previous similar message [ 2871.776507] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2871.791426] Lustre: Skipped 1 previous similar message [ 2872.264897] Lustre: server umount lustre-MDT0000 complete [ 2876.901918] LustreError: 81815:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2876.933724] LustreError: 81815:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2877.035081] LustreError: 81801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394610 with bad export cookie 17578195143498876659 [ 2877.045470] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2877.058974] LustreError: 81801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2877.470156] Lustre: server umount lustre-MDT0001 complete [ 2891.077228] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 2894.008757] Key type lgssc unregistered [ 2894.273034] LNet: 85385:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2894.287028] LNetError: 85385:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2894.305846] LNet: Removed LNI 192.168.202.105@tcp [ 2895.467366] Key type .llcrypt unregistered [ 2895.470669] Key type ._llcrypt unregistered [ 2925.381061] Key type ._llcrypt registered [ 2925.387035] Key type .llcrypt registered [ 2925.521107] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 2927.308483] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2927.405995] alg: No test for adler32 (adler32-zlib) [ 2928.403608] Lustre: Lustre: Build Version: 2.17.54_110_g2f98931 [ 2928.676425] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 2930.343195] Key type lgssc registered [ 2931.641377] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2940.291084] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2940.310271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2941.773302] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2946.644848] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2951.145467] LustreError: 86803:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2951.169616] LustreError: 86803:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2955.418407] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2955.839425] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2960.702606] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 2968.633941] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2975.501956] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2985.988055] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2986.479119] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2991.611408] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2996.119193] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 3004.781068] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3012.067577] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3012.072425] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3012.088235] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3016.162144] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3016.175672] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3016.191453] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3016.208191] Lustre: Skipped 1 previous similar message [ 3018.451990] Lustre: server umount lustre-OST0000 complete [ 3026.408122] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3026.410133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3026.419322] Lustre: Skipped 1 previous similar message [ 3026.429792] Lustre: Skipped 1 previous similar message [ 3031.529343] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3036.641728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3036.649084] Lustre: Skipped 2 previous similar messages [ 3037.151300] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3037.444984] Lustre: server umount lustre-MDT0000 complete [ 3041.765962] LustreError: 86802:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3041.787348] LustreError: 86802:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3042.034471] LustreError: 86783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394775 with bad export cookie 9788511990892242195 [ 3042.039159] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3042.050556] LustreError: 86783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3042.590713] Lustre: server umount lustre-MDT0001 complete [ 3058.083361] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 3061.113306] Key type lgssc unregistered [ 3061.479906] LNet: 90338:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3061.486832] LNetError: 90338:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3061.517512] LNet: Removed LNI 192.168.202.105@tcp [ 3062.675873] Key type .llcrypt unregistered [ 3062.678146] Key type ._llcrypt unregistered [ 3089.241920] Key type ._llcrypt registered [ 3089.244277] Key type .llcrypt registered [ 3089.348556] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 3090.277732] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3090.477378] alg: No test for adler32 (adler32-zlib) [ 3091.453775] Lustre: Lustre: Build Version: 2.17.54_110_g2f98931 [ 3091.637474] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 3093.287174] Key type lgssc registered [ 3094.326889] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3100.883279] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3100.905291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3102.372920] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3106.873542] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 3112.931753] LustreError: 91750:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3112.949367] LustreError: 91750:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3114.916113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3115.271689] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3119.951422] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 3127.564446] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3133.493897] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3141.068399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3141.429089] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3147.495807] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 3147.586218] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 3155.635987] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3176.429488] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3176.431809] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3176.440607] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3176.455261] Lustre: Skipped 1 previous similar message [ 3181.180249] Lustre: server umount lustre-OST0000 complete [ 3186.658750] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3186.674115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3186.684216] Lustre: Skipped 1 previous similar message [ 3186.691194] Lustre: Skipped 1 previous similar message [ 3191.779446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3191.798649] Lustre: Skipped 2 previous similar messages [ 3192.605699] Lustre: server umount lustre-MDT0000 complete [ 3196.899196] LustreError: 93039:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3196.919264] LustreError: 93039:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3196.986606] LustreError: 91732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394930 with bad export cookie 12442701495761419035 [ 3197.000212] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3197.000770] LustreError: 91732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3197.273743] Lustre: server umount lustre-MDT0001 complete [ 3211.583243] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 3216.688533] Key type lgssc unregistered [ 3217.183415] LNet: 95308:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3217.206815] LNetError: 95308:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3217.248704] LNet: Removed LNI 192.168.202.105@tcp [ 3218.755519] Key type .llcrypt unregistered [ 3218.767028] Key type ._llcrypt unregistered [ 3246.140973] Key type ._llcrypt registered [ 3246.143647] Key type .llcrypt registered [ 3246.273902] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 3247.668162] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3247.768560] alg: No test for adler32 (adler32-zlib) [ 3248.970332] Lustre: Lustre: Build Version: 2.17.54_110_g2f98931 [ 3249.272044] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 3250.953552] Key type lgssc registered [ 3252.469726] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3261.020131] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3261.040135] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3276.704399] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3281.824238] LustreError: 96699:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.105@tcp: failed processing log, type 4: rc = -110 [ 3308.447207] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3315.028123] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 3323.366372] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3331.090569] Lustre: server umount lustre-OST0000 complete [ 3340.357519] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 06:37:52 (1787395072) [ 3348.314304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3349.006908] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3353.351363] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 3354.082649] LustreError: 98337:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3354.103526] LustreError: 98337:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3358.183248] LustreError: 98338:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3362.289859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3362.525172] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3367.697598] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 3376.892870] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3383.275419] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3391.420447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3392.006903] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3394.042825] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3399.718116] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 3409.369719] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3422.689470] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3422.692375] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3422.711144] Lustre: Skipped 1 previous similar message [ 3422.713095] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3422.719448] Lustre: Skipped 1 previous similar message [ 3427.712035] Lustre: server umount lustre-OST0000 complete [ 3432.930237] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3432.931651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3432.946770] Lustre: Skipped 1 previous similar message [ 3438.049552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3438.060675] Lustre: Skipped 2 previous similar messages [ 3438.507139] Lustre: server umount lustre-MDT0000 complete [ 3442.161414] LustreError: 98332:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3442.196140] LustreError: 98332:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3443.831704] LustreError: 98319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395177 with bad export cookie 12768579353679693139 [ 3443.855383] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3443.856652] LustreError: 98319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3444.667268] Lustre: server umount lustre-MDT0001 complete [ 3459.525507] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 3462.922187] Key type lgssc unregistered [ 3463.271762] LNet: 102031:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3463.296755] LNetError: 102031:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3464.364341] LNet: Removed LNI 192.168.202.105@tcp [ 3465.462310] Key type .llcrypt unregistered [ 3465.468970] Key type ._llcrypt unregistered [ 3516.325169] Key type ._llcrypt registered [ 3516.330073] Key type .llcrypt registered [ 3516.431673] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 3517.398966] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3517.430676] alg: No test for adler32 (adler32-zlib) [ 3518.520586] Lustre: Lustre: Build Version: 2.17.54_110_g2f98931 [ 3518.921251] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 3520.703579] Key type lgssc registered [ 3522.148180] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3531.890968] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3531.938532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3533.558544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3538.417184] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 3544.034092] LustreError: 104221:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3544.065304] LustreError: 104221:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3549.163843] LustreError: 104220:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3549.541664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3550.005440] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3556.120957] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 3564.290747] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3572.378446] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3580.696425] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3581.285560] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3589.315851] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 3601.291337] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3603.902701] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3618.789827] Lustre: server umount lustre-OST0000 complete [ 3624.420622] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3624.422207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3624.441898] Lustre: Skipped 1 previous similar message [ 3624.473193] Lustre: Skipped 1 previous similar message [ 3629.538774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3629.546386] Lustre: Skipped 1 previous similar message [ 3633.692984] Lustre: server umount lustre-MDT0000 complete [ 3634.668039] LustreError: 106714:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3634.686785] LustreError: 106714:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3635.682326] Lustre: 103531:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787395353/real 1787395353] req@00000000ca5bae2c x1874219763793152/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787395369 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3635.729485] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3636.767451] Lustre: 103529:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787395354/real 1787395354] req@000000002c15857f x1874219763793536/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787395370 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3637.630914] LustreError: 104200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395371 with bad export cookie 9174149758899119070 [ 3637.644324] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3638.097673] Lustre: server umount lustre-MDT0001 complete [ 3650.532465] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 3653.475759] Key type lgssc unregistered [ 3653.753739] LNet: 107774:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3653.759596] LNetError: 107774:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3653.781886] LNet: Removed LNI 192.168.202.105@tcp [ 3654.628275] Key type .llcrypt unregistered [ 3654.632133] Key type ._llcrypt unregistered [ 3676.007986] Key type ._llcrypt registered [ 3676.010327] Key type .llcrypt registered [ 3676.096875] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 3677.046425] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3677.069907] alg: No test for adler32 (adler32-zlib) [ 3678.152946] Lustre: Lustre: Build Version: 2.17.54_110_g2f98931 [ 3678.471152] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 3680.108236] Key type lgssc registered [ 3681.125414] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3690.175792] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3690.198727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3705.887962] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3711.007784] LustreError: 109016:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.105@tcp: failed processing log, type 4: rc = -110 [ 3737.633119] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3744.197316] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 3752.341719] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3759.245610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3759.362109] LustreError: 108352:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000094287b90 x1874219931535616/t0(0) o250->MGC192.168.202.105@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3759.574220] LustreError: MGC192.168.202.105@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3759.585159] LustreError: 110040:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3759.592912] LustreError: 110040:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3759.605314] LustreError: 110040:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3759.721898] Lustre: server umount lustre-MDT0000 complete [ 3759.725831] LustreError: 110040:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3781.087970] Lustre: 108353:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787395498/real 1787395498] req@00000000c93799aa x1874219931536768/t0(0) o400->MGC192.168.202.105@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787395514 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3781.105937] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3781.120757] LustreError: 109028:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3801.668410] Lustre: server umount lustre-OST0000 complete [ 3810.617739] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_hostid [ 3820.785261] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 3861.033613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3861.303723] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3861.350518] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3861.456937] Lustre: lustre-MDT0000: new disk, initializing [ 3861.539594] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3861.554511] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3865.883462] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 3876.318326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3876.468325] Lustre: 112835:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3876.534267] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3876.538535] Lustre: Skipped 1 previous similar message [ 3876.611681] Lustre: lustre-MDT0001: new disk, initializing [ 3876.711124] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3876.765740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3876.784419] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3881.250776] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 3890.659092] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3896.399602] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3903.210752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3903.422886] Lustre: lustre-OST0000: new disk, initializing [ 3903.427883] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3903.435563] Lustre: 114805:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3903.506211] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3910.299820] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 3911.745781] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3911.759442] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3911.863070] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3919.726803] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3925.906911] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3926.246661] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3931.440552] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3931.627181] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3937.768040] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3937.773248] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3937.780371] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3939.945767] Lustre: server umount lustre-OST0000 complete [ 3948.001184] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3948.019259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3948.035603] Lustre: Skipped 2 previous similar messages [ 3948.048181] Lustre: Skipped 1 previous similar message [ 3950.643420] Lustre: server umount lustre-MDT0000 complete [ 3957.135668] LustreError: 112827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395691 with bad export cookie 4754072247187038722 [ 3957.161749] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3957.165612] LustreError: 112827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3958.013518] Lustre: server umount lustre-MDT0001 complete [ 3969.661676] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 06:48:21 (1787395701) [ 3973.164806] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3990.719672] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_hostid [ 4000.957229] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 4036.047986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4036.404776] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4036.430816] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4036.533830] Lustre: lustre-MDT0000: new disk, initializing [ 4036.676827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4036.721418] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4041.248578] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4053.244290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4053.371117] Lustre: 119708:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4053.428959] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4053.432476] Lustre: Skipped 1 previous similar message [ 4053.492338] Lustre: lustre-MDT0001: new disk, initializing [ 4053.573458] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4053.628704] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4053.638217] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4058.585561] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4068.800892] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4075.373221] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4083.739477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4084.152896] Lustre: lustre-OST0000: new disk, initializing [ 4084.165905] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4084.182341] Lustre: 121676:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4084.288984] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4090.471286] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4090.485476] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4090.619066] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4091.753499] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4101.406821] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4106.355391] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4106.718921] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4111.258558] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4111.565252] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4116.450108] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4116.473029] LustreError: Skipped 1 previous similar message [ 4116.476877] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4116.494031] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4121.568910] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4121.571758] Lustre: Skipped 1 previous similar message [ 4126.692076] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4126.707324] Lustre: Skipped 1 previous similar message [ 4128.223732] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4128.489275] Lustre: server umount lustre-OST0000 complete [ 4135.395120] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4135.398892] LustreError: Skipped 1 previous similar message [ 4135.401499] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4135.409903] Lustre: Skipped 1 previous similar message [ 4135.414129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4135.418183] Lustre: Skipped 1 previous similar message [ 4139.139906] Lustre: server umount lustre-MDT0000 complete [ 4142.050985] LustreError: 119718:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4142.072507] LustreError: 119718:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 4143.526554] LustreError: 121675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395877 with bad export cookie 4754072247187039604 [ 4143.541409] LustreError: 121675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4143.542960] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4144.051637] Lustre: server umount lustre-MDT0001 complete [ 4153.433759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4153.986637] LustreError: 123897:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4154.114573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4159.251350] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4159.472969] LustreError: 123897:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4159.489731] LustreError: 123897:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4163.558530] LustreError: 123898:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4168.673752] LustreError: 123897:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4169.052062] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4175.912802] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4185.172803] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4193.379514] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4201.189198] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4208.575822] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4218.322966] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4230.702878] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 4235.745952] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4235.758489] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4235.783157] Lustre: Skipped 1 previous similar message [ 4235.789057] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4235.809266] Lustre: Skipped 2 previous similar messages [ 4237.896719] Lustre: server umount lustre-OST0000 complete [ 4244.974467] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4252.226748] Lustre: server umount lustre-MDT0000 complete [ 4254.204323] LustreError: 125166:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4254.234332] LustreError: 125166:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4256.442546] LustreError: 123878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395990 with bad export cookie 4754072247187040486 [ 4256.471784] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4256.981616] Lustre: server umount lustre-MDT0001 complete [ 4271.068537] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 4274.826868] Key type lgssc unregistered [ 4275.139354] LNet: 127529:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4275.145250] LNetError: 127529:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4275.173650] LNet: Removed LNI 192.168.202.105@tcp [ 4276.507138] Key type .llcrypt unregistered [ 4276.511942] Key type ._llcrypt unregistered [ 4301.490600] Key type ._llcrypt registered [ 4301.492209] Key type .llcrypt registered [ 4301.622256] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_hostid [ 4319.389414] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 4319.796859] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4319.842806] alg: No test for adler32 (adler32-zlib) [ 4320.800467] Lustre: Lustre: Build Version: 2.17.54_110_g2f98931 [ 4321.177314] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 4322.911396] Key type lgssc registered [ 4325.033905] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4368.379645] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4368.438170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4369.689194] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4369.717892] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4369.807135] Lustre: lustre-MDT0000: new disk, initializing [ 4369.868853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4369.885785] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4374.791074] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4385.598374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4385.783409] Lustre: 130683:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4385.829253] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4385.832193] Lustre: Skipped 1 previous similar message [ 4385.932212] Lustre: lustre-MDT0001: new disk, initializing [ 4386.024942] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4386.077265] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4386.094713] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4390.312894] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4400.161696] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4406.355718] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4413.579499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4413.857261] Lustre: lustre-OST0000: new disk, initializing [ 4413.863374] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4413.873471] Lustre: 132648:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4413.970353] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4421.012714] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4421.787724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4421.820676] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4422.031784] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4430.232859] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4435.049177] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4435.341288] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4441.046150] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4441.299734] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4447.713346] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4447.722337] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4447.763061] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4447.779808] Lustre: Skipped 1 previous similar message [ 4449.944325] Lustre: server umount lustre-OST0000 complete [ 4457.954442] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4457.956069] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4457.966027] LustreError: Skipped 1 previous similar message [ 4457.966822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4457.972218] Lustre: Skipped 2 previous similar messages [ 4463.073697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4463.081677] Lustre: Skipped 2 previous similar messages [ 4467.172393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4467.177924] Lustre: Skipped 1 previous similar message [ 4468.191589] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4468.522575] Lustre: server umount lustre-MDT0000 complete [ 4472.672927] LustreError: 130676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787396206 with bad export cookie 7681537738674795604 [ 4472.674585] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4472.685375] LustreError: 130676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4473.031617] Lustre: server umount lustre-MDT0001 complete [ 4482.292144] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 06:56:54 (1787396214) [ 4490.027369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4490.579117] LustreError: 135064:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4490.601717] LustreError: 135064:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4490.654937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4495.522739] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4495.851611] LustreError: 135065:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4500.960085] LustreError: 135064:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4504.608045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4505.028972] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4509.592884] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4518.476787] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4525.108262] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4527.381668] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4527.385047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4527.400172] Lustre: Skipped 1 previous similar message [ 4527.419645] Lustre: Skipped 1 previous similar message [ 4534.058147] Lustre: server umount lustre-MDT0000 complete [ 4534.752723] LustreError: 135064:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4534.787581] LustreError: 135064:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4538.465467] LustreError: 135046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787396272 with bad export cookie 7681537738674796486 [ 4538.472864] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4538.485296] LustreError: 135046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4538.818700] Lustre: server umount lustre-MDT0001 complete [ 4547.676477] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 06:57:59 (1787396279) [ 4557.167921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4573.023773] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4578.143602] LustreError: 137482:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.105@tcp: failed processing log, type 4: rc = -110 [ 4603.939512] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4611.417240] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4619.636261] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4621.823904] Lustre: server umount lustre-OST0000 complete [ 4630.491217] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 06:59:22 (1787396362) [ 4637.451808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4637.934205] LustreError: 138929:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4638.045377] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4644.196183] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4647.401164] LustreError: 138930:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4647.454231] LustreError: 138930:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4656.347408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4661.882254] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4671.074449] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4679.049698] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4687.870245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4688.501141] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4688.507955] Lustre: Skipped 1 previous similar message [ 4695.731040] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4704.754491] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4718.562834] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4718.579169] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4718.614658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4718.619838] Lustre: Skipped 2 previous similar messages [ 4723.030663] Lustre: server umount lustre-MDT0000 complete [ 4724.710574] LustreError: 138930:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4724.730778] LustreError: 138930:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4728.188513] LustreError: 138910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787396462 with bad export cookie 7681537738674797144 [ 4728.190763] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4728.213325] LustreError: 138910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4729.232352] Lustre: server umount lustre-MDT0001 complete [ 4745.331834] Lustre: server umount lustre-OST0000 complete [ 4756.367808] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 07:01:28 (1787396488) [ 4764.180722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4764.783506] LustreError: 142596:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4764.921618] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4769.491220] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4779.253238] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4786.482114] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4795.946931] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4803.372991] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4811.981413] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4819.986880] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4829.944505] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4835.084215] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4837.109869] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4837.454542] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4843.414978] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4843.659589] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4850.680097] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4850.680699] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4850.692228] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4850.692237] Lustre: Skipped 3 previous similar messages [ 4850.705749] Lustre: Skipped 3 previous similar messages [ 4850.728837] LustreError: Skipped 1 previous similar message [ 4860.383244] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4860.691975] Lustre: server umount lustre-OST0000 complete [ 4866.025358] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4879.329859] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4879.688342] Lustre: server umount lustre-MDT0000 complete [ 4880.401109] LustreError: 142591:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4880.433366] LustreError: 142591:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 4884.495595] LustreError: 142578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787396618 with bad export cookie 7681537738674798215 [ 4884.507982] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4884.511423] LustreError: 142578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4885.501362] Lustre: server umount lustre-MDT0001 complete [ 4897.198785] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 07:03:49 (1787396629) [ 4904.723978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4920.287784] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4925.410358] LustreError: 146763:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.105@tcp: failed processing log, type 4: rc = -110 [ 4951.135827] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4951.138335] Lustre: Skipped 2 previous similar messages [ 4958.087815] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4967.588607] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4977.216026] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4978.144446] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4983.465586] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 4992.992901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4998.580411] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5008.552811] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5015.509916] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5021.427671] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5021.732950] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5027.605960] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5027.919606] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5034.467155] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5034.470016] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5034.470712] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5034.480883] Lustre: Skipped 2 previous similar messages [ 5034.505100] Lustre: Skipped 11 previous similar messages [ 5036.772669] Lustre: server umount lustre-MDT0000 complete [ 5039.594620] LustreError: 147801:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5039.631581] LustreError: 147801:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 5041.208396] LustreError: 146769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787396775 with bad export cookie 7681537738674799244 [ 5041.255686] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5041.751581] Lustre: server umount lustre-MDT0001 complete [ 5056.470429] Lustre: server umount lustre-OST0000 complete [ 5066.797630] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 07:06:38 (1787396798) [ 5076.233369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5091.744689] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5096.863486] LustreError: 150961:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.105@tcp: failed processing log, type 4: rc = -110 [ 5130.342374] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5139.143699] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5146.713251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5147.585474] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 5152.620514] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5163.801903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5169.109585] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5178.125898] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5186.053940] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5193.277893] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5193.583874] Lustre: lustre-OST0001: new disk, initializing [ 5193.593552] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5193.607887] Lustre: 153836:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5200.032965] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5200.048992] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5200.175860] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5201.263959] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5212.990775] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5218.552964] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5218.827152] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5224.079135] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5224.403750] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5229.538744] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5229.554778] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5229.567261] Lustre: Skipped 1 previous similar message [ 5229.572760] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5233.426133] Lustre: server umount lustre-OST0000 complete [ 5252.063806] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5252.427073] Lustre: server umount lustre-OST0001 complete [ 5261.796979] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5261.810659] LustreError: Skipped 3 previous similar messages [ 5263.519915] Lustre: server umount lustre-MDT0000 complete [ 5267.487785] LustreError: 150969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787397001 with bad export cookie 7681537738674800196 [ 5267.492538] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5267.497870] LustreError: 150969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 5268.144728] Lustre: server umount lustre-MDT0001 complete [ 5294.955945] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 07:10:27 (1787397027) [ 5296.933658] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 5298.769170] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 5300.749239] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 07:10:33 (1787397033) [ 5307.526932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5307.775982] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5307.815633] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5307.973748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5307.980955] Lustre: Skipped 6 previous similar messages [ 5311.538591] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5321.434398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5321.597471] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5321.653519] Lustre: 157404:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5327.721425] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5338.070488] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5345.195365] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5352.311938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5352.503818] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5358.128995] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5361.202814] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 5365.510156] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5370.862668] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5371.127820] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5377.195326] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5377.459656] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5381.614751] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5381.619413] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5381.619643] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5381.628858] Lustre: Skipped 6 previous similar messages [ 5381.667067] Lustre: Skipped 11 previous similar messages [ 5394.401202] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5394.620322] Lustre: server umount lustre-OST0000 complete [ 5420.459896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5422.131084] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5422.144488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5422.148782] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5422.184185] Lustre: Skipped 1 previous similar message [ 5428.998806] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5437.749374] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5443.950763] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5444.248787] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5450.540360] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5450.853914] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5458.858814] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8e68888f5000.ost_server_uuid 50 [ 5461.201681] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8e68888f5000.ost_server_uuid in IDLE state after 0 sec [ 5461.484275] LustreError: 157412:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.202.5@tcp) returned error from blocking AST (req@00000000e28fb5ec x1874220606056832 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000d6afb517/0x6a9a4ddff2780f9e lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.5@tcp remote: 0xb3d7fdd570cf63e9 expref: 5 pid: 159030 timeout: 5561 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5461.529445] LustreError: lustre-MDT0000: A client on nid 192.168.202.5@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5461.550791] LustreError: 157400:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.202.5@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000d6afb517/0x6a9a4ddff2780f9e lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.5@tcp remote: 0xb3d7fdd570cf63e9 expref: 6 pid: 159030 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5467.126200] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5467.140634] LustreError: Skipped 1 previous similar message [ 5473.253812] LustreError: 161035:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5473.284429] LustreError: 161035:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 17 previous similar messages [ 5473.451898] Lustre: server umount lustre-OST0000 complete [ 5483.959449] Lustre: server umount lustre-MDT0000 complete [ 5487.959927] LustreError: 157397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787397221 with bad export cookie 7681537738674801379 [ 5487.968934] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5487.980658] LustreError: 157397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5494.406196] Lustre: server umount lustre-MDT0001 complete [ 5506.965856] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 5510.851468] Key type lgssc unregistered [ 5511.175540] LNet: 163522:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5511.197860] LNetError: 163522:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5511.230202] LNet: Removed LNI 192.168.202.105@tcp [ 5512.334878] Key type .llcrypt unregistered [ 5512.339630] Key type ._llcrypt unregistered [ 5543.145401] Key type ._llcrypt registered [ 5543.147639] Key type .llcrypt registered [ 5543.269351] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 5544.440993] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5544.608305] alg: No test for adler32 (adler32-zlib) [ 5545.648829] Lustre: Lustre: Build Version: 2.17.54_110_g2f98931 [ 5545.837242] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 5547.504158] Key type lgssc registered [ 5548.886546] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5558.400846] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5558.432167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5560.065979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5564.831562] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5569.506974] LustreError: 164941:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5569.515431] LustreError: 164941:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5573.717461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5574.082433] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5578.771591] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5586.535973] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5593.872241] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5601.912612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5602.334750] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5604.408971] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5608.858697] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5618.010172] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5624.192897] Lustre: Failing over lustre-MDT0000 [ 5624.656831] Lustre: server umount lustre-MDT0000 complete [ 5625.273431] LustreError: 164935:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5625.312463] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5625.329384] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5628.897801] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5628.899459] LustreError: 167146:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5628.938250] Lustre: Skipped 1 previous similar message [ 5628.986081] LustreError: 167146:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5634.018599] LustreError: 164940:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5634.050256] LustreError: 164940:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5636.838524] LustreError: 164923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.5@tcp arrived at 1787397370 with bad export cookie 17422072027129336445 [ 5639.141272] LustreError: 164941:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5639.156283] LustreError: 164941:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5644.255164] Lustre: 164250:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787397362/real 1787397362] req@0000000061f9d360 x1874221889252480/t0(0) o400->MGC192.168.202.105@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787397378 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5644.308661] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5645.025311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5649.384107] LustreError: 164936:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5649.405993] LustreError: 164936:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5654.504670] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xf1c7a233dc5052df [ 5654.518818] Lustre: MGC192.168.202.105@tcp: Connection restored to 0@lo (at 0@lo) [ 5654.841139] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5654.942591] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5659.593228] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5660.130175] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5660.140631] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5662.186130] Lustre: lustre-MDT0000: Denying connection for new client d229f831-9a15-4da2-9ad9-f2b140f02c5a (at 192.168.202.5@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 5667.235841] Lustre: lustre-MDT0000: Denying connection for new client d229f831-9a15-4da2-9ad9-f2b140f02c5a (at 192.168.202.5@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:03 [ 5674.984721] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5675.001109] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5675.028020] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5680.609919] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5680.623995] Lustre: Skipped 3 previous similar messages [ 5685.215286] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5685.399287] Lustre: server umount lustre-OST0000 complete [ 5689.562592] LustreError: 168674:0:(ldlm_lib.c:2990:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5689.575265] Lustre: 168023:0:(ldlm_lib.c:2390:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5689.582337] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5689.594609] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5689.606672] LustreError: 168023:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000c9ee93b0 x1874221889272960/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5689.617693] LustreError: 168023:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5689.626836] LustreError: 168023:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5689.654876] Lustre: lustre-MDT0000: Recovery over after 0:29, of 2 clients 0 recovered and 2 were evicted. [ 5690.024120] Lustre: server umount lustre-MDT0000 complete [ 5690.858446] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5690.869921] Lustre: Skipped 1 previous similar message [ 5690.875357] LustreError: 164937:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5690.898788] LustreError: 164937:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5694.639890] LustreError: 164922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787397428 with bad export cookie 17422072027129336543 [ 5694.655235] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5694.658589] LustreError: 164922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5695.008937] Lustre: server umount lustre-MDT0001 complete [ 5707.594538] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 5710.263422] Key type lgssc unregistered [ 5710.551493] LNet: 169428:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5710.559851] LNetError: 169428:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5710.584327] LNet: Removed LNI 192.168.202.105@tcp [ 5711.450884] Key type .llcrypt unregistered [ 5711.453472] Key type ._llcrypt unregistered [ 5741.000758] Key type ._llcrypt registered [ 5741.003926] Key type .llcrypt registered [ 5741.164681] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 5742.438038] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5742.688603] alg: No test for adler32 (adler32-zlib) [ 5743.933465] Lustre: Lustre: Build Version: 2.17.54_110_g2f98931 [ 5744.240882] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 5745.991567] Key type lgssc registered [ 5747.002932] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5755.458022] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5755.494052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5757.038959] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5762.864401] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5766.625203] LustreError: 170852:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5766.638093] LustreError: 170852:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5771.750729] LustreError: 170853:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5772.982512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5773.302496] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5777.775740] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5786.150398] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5792.432372] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5799.290617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5799.640547] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5801.737737] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5806.274674] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5815.243179] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5822.434177] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5822.442220] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5822.458296] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5824.499729] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5824.504608] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5824.509925] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5828.066958] Lustre: server umount lustre-OST0000 complete [ 5834.720658] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5834.724058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5834.739585] Lustre: Skipped 1 previous similar message [ 5834.747907] Lustre: Skipped 2 previous similar messages [ 5839.053333] Lustre: server umount lustre-MDT0000 complete [ 5839.841821] LustreError: 170849:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5839.871219] LustreError: 170849:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5842.918591] LustreError: 170833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787397576 with bad export cookie 12854087456935669024 [ 5842.935324] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5843.208291] Lustre: server umount lustre-MDT0001 complete [ 5855.661821] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 5859.709660] Key type lgssc unregistered [ 5860.300172] LNet: 174383:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5860.307837] LNetError: 174383:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5860.378163] LNet: Removed LNI 192.168.202.105@tcp [ 5861.372669] Key type .llcrypt unregistered [ 5861.377104] Key type ._llcrypt unregistered [ 5907.851360] Key type ._llcrypt registered [ 5907.862495] Key type .llcrypt registered [ 5907.984671] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 5909.532269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5909.593233] alg: No test for adler32 (adler32-zlib) [ 5910.692666] Lustre: Lustre: Build Version: 2.17.54_110_g2f98931 [ 5911.034443] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 5912.751319] Key type lgssc registered [ 5914.195792] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5923.490541] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5923.524759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5925.216376] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5930.196184] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5935.615428] LustreError: 176090:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5935.634586] LustreError: 176090:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5940.003843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5940.615061] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5946.629197] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5955.561893] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5962.817582] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5971.273929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5971.637103] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5980.699721] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 5989.956195] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5995.189528] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 6005.732938] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6005.736985] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6005.746142] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6005.775448] Lustre: Skipped 1 previous similar message [ 6009.511544] Lustre: server umount lustre-OST0000 complete [ 6015.968628] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6015.973284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6015.987145] Lustre: Skipped 1 previous similar message [ 6016.009226] Lustre: Skipped 2 previous similar messages [ 6021.089749] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6021.100320] Lustre: Skipped 1 previous similar message [ 6025.185984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6025.195447] Lustre: Skipped 1 previous similar message [ 6028.255306] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6028.437733] Lustre: server umount lustre-MDT0000 complete [ 6030.308693] LustreError: 176090:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6030.346231] LustreError: 176090:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6033.148522] LustreError: 176069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787397767 with bad export cookie 8180125545024921953 [ 6033.154110] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6033.161714] LustreError: 176069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6033.824380] Lustre: server umount lustre-MDT0001 complete [ 6048.147355] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 6051.436371] Key type lgssc unregistered [ 6051.714710] LNet: 179646:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6051.721376] LNetError: 179646:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6051.750280] LNet: Removed LNI 192.168.202.105@tcp [ 6052.883113] Key type .llcrypt unregistered [ 6052.887808] Key type ._llcrypt unregistered [ 6081.963352] Key type ._llcrypt registered [ 6081.966269] Key type .llcrypt registered [ 6082.070656] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 6083.503684] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6083.529048] alg: No test for adler32 (adler32-zlib) [ 6084.601846] Lustre: Lustre: Build Version: 2.17.54_110_g2f98931 [ 6084.939408] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 6086.735696] Key type lgssc registered [ 6087.968702] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6100.141930] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6100.157995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6101.548904] Lustre: *** cfs_fail_loc=135, val=0*** [ 6101.606026] LustreError: 181184:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 6101.611982] LustreError: 181184:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.105@tcp: cfg command failed: rc = -2 [ 6101.620839] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 6101.620839] [ 6101.631711] LustreError: MGC192.168.202.105@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 6101.654918] LustreError: 181145:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 6101.663550] LustreError: 181145:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 6101.668476] LustreError: 181145:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 6101.673227] LustreError: 181145:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 6101.741098] Lustre: server umount lustre-MDT0000 complete [ 6101.747651] LustreError: 181145:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 6108.611921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6109.024715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6113.700311] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6118.370273] LustreError: 181556:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6118.381562] LustreError: 181556:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6123.450845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6123.492625] LustreError: 181557:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6123.908870] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6129.085923] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6137.805515] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6144.309521] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6148.578823] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6148.584396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6148.588346] Lustre: Skipped 1 previous similar message [ 6148.607807] Lustre: Skipped 1 previous similar message [ 6152.820205] Lustre: server umount lustre-MDT0000 complete [ 6153.706218] LustreError: 181556:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6153.727347] LustreError: 181556:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6157.059968] LustreError: 181538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787397890 with bad export cookie 18230278267621097590 [ 6157.085191] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6157.718352] Lustre: server umount lustre-MDT0001 complete [ 6171.812062] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 6174.755458] Key type lgssc unregistered [ 6175.150743] LNet: 183876:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6175.159283] LNetError: 183876:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6175.186067] LNet: Removed LNI 192.168.202.105@tcp [ 6176.237489] Key type .llcrypt unregistered [ 6176.246970] Key type ._llcrypt unregistered [ 6202.187632] Key type ._llcrypt registered [ 6202.190184] Key type .llcrypt registered [ 6202.295835] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 6202.552584] Key type .llcrypt unregistered [ 6202.554918] Key type ._llcrypt unregistered [ 6224.019085] Key type ._llcrypt registered [ 6224.020693] Key type .llcrypt registered [ 6224.202566] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 6225.463869] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6225.602070] alg: No test for adler32 (adler32-zlib) [ 6226.706864] Lustre: Lustre: Build Version: 2.17.54_110_g2f98931 [ 6226.911229] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 6228.559719] Key type lgssc registered [ 6230.000735] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6238.757651] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6238.777405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6254.367897] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6259.487555] LustreError: 186012:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.105@tcp: failed processing log, type 4: rc = -110 [ 6286.239212] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6293.201381] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6302.381518] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6310.576705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6311.410674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6311.574183] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 6316.516929] LustreError: 186037:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6316.530015] LustreError: 186037:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6316.669487] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6320.664071] LustreError: 187051:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6320.717508] LustreError: 187051:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6325.732481] LustreError: 186038:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6325.748518] LustreError: 186038:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6328.235370] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6328.653180] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6333.619929] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6342.716802] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6350.146434] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6354.018956] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6367.715550] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6367.734574] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6367.746546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6369.764186] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6369.769754] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6369.773474] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6369.795704] Lustre: Skipped 1 previous similar message [ 6372.060096] Lustre: server umount lustre-OST0000 complete [ 6380.003989] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6380.006357] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6380.064800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6382.836512] Lustre: server umount lustre-MDT0000 complete [ 6383.021757] LustreError: 187055:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6386.919588] LustreError: 188673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787398120 with bad export cookie 13815069074147850683 [ 6386.926842] LustreError: 188673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6386.930113] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6387.345458] Lustre: server umount lustre-MDT0001 complete [ 6401.152570] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 6404.763772] Key type lgssc unregistered [ 6405.075106] LNet: 189979:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6405.085662] LNetError: 189979:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6406.119447] LNet: Removed LNI 192.168.202.105@tcp [ 6407.157589] Key type .llcrypt unregistered [ 6407.163711] Key type ._llcrypt unregistered [ 6434.334631] Key type ._llcrypt registered [ 6434.336776] Key type .llcrypt registered [ 6434.443149] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 6436.022588] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6436.099055] alg: No test for adler32 (adler32-zlib) [ 6437.352256] Lustre: Lustre: Build Version: 2.17.54_110_g2f98931 [ 6437.754540] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 6439.431946] Key type lgssc registered [ 6440.869605] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6449.728473] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6449.745895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6451.207311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6455.592120] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6461.413042] LustreError: 191395:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6461.435573] LustreError: 191395:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6465.167819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6465.612093] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6470.025135] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6478.005454] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6484.673647] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6492.389151] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6492.905847] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6497.018474] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 6500.898734] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6509.754708] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6515.868131] Lustre: Failing over lustre-MDT0000 [ 6516.286758] Lustre: server umount lustre-MDT0000 complete [ 6516.705537] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6516.712860] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6516.728769] LustreError: 191395:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6520.808403] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6520.808775] LustreError: 191396:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6520.861720] LustreError: 191396:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6525.934130] LustreError: 191391:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6525.953989] LustreError: 191391:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6531.052255] LustreError: 191395:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6531.067818] LustreError: 191395:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6536.159755] Lustre: 190707:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787398254/real 1787398254] req@0000000021c76fa3 x1874222824582016/t0(0) o400->MGC192.168.202.105@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787398270 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6536.216539] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6536.302704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6541.284931] LustreError: 191395:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6541.304973] LustreError: 191395:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 6546.404093] LustreError: 190706:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000005cfb33d6 x1874222824587392/t0(0) o250->MGC192.168.202.105@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6546.860705] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6546.909675] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6549.418668] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6551.532493] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6552.042278] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6552.083358] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 6552.160705] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 6557.203086] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6564.587913] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6572.518316] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6572.532980] Lustre: Skipped 1 previous similar message [ 6572.539339] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6577.121071] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6577.127451] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6577.141075] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6582.767784] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6582.777025] Lustre: Skipped 2 previous similar messages [ 6586.335137] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6586.601369] Lustre: server umount lustre-OST0000 complete [ 6593.004155] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6593.009417] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6593.019570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6593.055646] Lustre: Skipped 1 previous similar message [ 6598.114706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6598.122065] Lustre: Skipped 2 previous similar messages [ 6605.279160] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6605.444729] Lustre: server umount lustre-MDT0000 complete [ 6607.330432] LustreError: 193888:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6607.353579] LustreError: 193888:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 6609.944422] LustreError: 193240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787398343 with bad export cookie 3504003431823450308 [ 6609.946220] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6609.956216] LustreError: 193240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6610.281622] Lustre: server umount lustre-MDT0001 complete [ 6622.270296] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 6625.740545] Key type lgssc unregistered [ 6626.021842] LNet: 196239:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6626.025504] LNetError: 196239:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6627.046283] LNet: Removed LNI 192.168.202.105@tcp [ 6628.026187] Key type .llcrypt unregistered [ 6628.028446] Key type ._llcrypt unregistered [ 6658.790709] Key type ._llcrypt registered [ 6658.792371] Key type .llcrypt registered [ 6658.963288] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 6660.555804] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6660.575732] alg: No test for adler32 (adler32-zlib) [ 6661.613797] Lustre: Lustre: Build Version: 2.17.54_110_g2f98931 [ 6661.849293] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 6663.535622] Key type lgssc registered [ 6664.545916] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6672.691511] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6672.713870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6674.366865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6679.383625] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6684.660778] LustreError: 197654:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6684.689469] LustreError: 197654:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6687.311026] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6687.776307] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6691.970546] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6701.667143] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6708.063612] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6715.169376] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6715.444837] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6719.548111] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6721.631317] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6729.657486] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6733.271614] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6740.442985] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6755.280363] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6768.609631] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6768.622189] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6768.624258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6768.643095] Lustre: Skipped 1 previous similar message [ 6771.833186] Lustre: server umount lustre-OST0000 complete [ 6778.851020] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6778.876717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6778.888165] Lustre: Skipped 2 previous similar messages [ 6782.729592] Lustre: server umount lustre-MDT0000 complete [ 6783.979112] LustreError: 197655:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6786.726636] LustreError: 197634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787398520 with bad export cookie 13059121154159195561 [ 6786.728868] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6786.736674] LustreError: 197634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6787.163794] Lustre: server umount lustre-MDT0001 complete [ 6800.786500] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 6804.534085] Key type lgssc unregistered [ 6804.756533] LNet: 201499:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6804.767310] LNetError: 201499:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6804.802442] LNet: Removed LNI 192.168.202.105@tcp [ 6806.003610] Key type .llcrypt unregistered [ 6806.015195] Key type ._llcrypt unregistered [ 6827.712469] Key type ._llcrypt registered [ 6827.715715] Key type .llcrypt registered [ 6827.814984] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 6829.251417] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6829.275232] alg: No test for adler32 (adler32-zlib) [ 6830.503589] Lustre: Lustre: Build Version: 2.17.54_110_g2f98931 [ 6830.858550] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 6832.607264] Key type lgssc registered [ 6833.927645] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6841.582154] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6841.601853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6843.092130] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6847.828911] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6852.591853] LustreError: 202768:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6852.614339] LustreError: 202768:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6857.526875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6857.697694] LustreError: 202769:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6857.970130] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6862.564274] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6870.659347] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6876.222309] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6883.981038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6884.339289] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6887.426462] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6890.896970] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 6898.997809] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6905.684413] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6914.659471] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6929.631736] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6936.008883] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6948.832378] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6948.838456] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6948.847732] LustreError: Skipped 1 previous similar message [ 6948.851461] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6948.872064] Lustre: Skipped 1 previous similar message [ 6952.540560] Lustre: server umount lustre-OST0000 complete [ 6959.072679] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6959.090393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6959.095056] Lustre: Skipped 1 previous similar message [ 6959.123524] Lustre: Skipped 2 previous similar messages [ 6963.914135] Lustre: server umount lustre-MDT0000 complete [ 6964.197895] LustreError: 203785:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6968.307273] LustreError: 202764:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6968.330138] LustreError: 202764:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6968.354395] LustreError: 202751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787398702 with bad export cookie 11478583544717164216 [ 6968.362328] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6968.365569] LustreError: 202751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6968.612303] Lustre: server umount lustre-MDT0001 complete [ 6980.341580] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 6983.399560] Key type lgssc unregistered [ 6983.758964] LNet: 207790:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6983.766974] LNetError: 207790:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6984.812553] LNet: Removed LNI 192.168.202.105@tcp [ 6985.833115] Key type .llcrypt unregistered [ 6985.841876] Key type ._llcrypt unregistered [ 7011.266780] Key type ._llcrypt registered [ 7011.270704] Key type .llcrypt registered [ 7011.382806] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 7012.414881] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7012.440996] alg: No test for adler32 (adler32-zlib) [ 7013.530028] Lustre: Lustre: Build Version: 2.17.54_110_g2f98931 [ 7013.737995] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 7015.431209] Key type lgssc registered [ 7016.466746] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7024.685834] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7024.700508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7026.278072] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7031.661772] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7036.391198] LustreError: 209204:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7036.439058] LustreError: 209204:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7040.534046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7040.869590] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7045.218508] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7053.549077] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7060.960980] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7070.033891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7070.576585] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7071.624141] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 7079.288635] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7090.038924] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7106.533128] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7106.534301] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7106.539575] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7106.544303] Lustre: Skipped 1 previous similar message [ 7111.650425] LustreError: 211026:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7111.786186] Lustre: server umount lustre-OST0000 complete [ 7116.770178] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7116.787502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7116.796958] Lustre: Skipped 1 previous similar message [ 7116.810082] Lustre: Skipped 2 previous similar messages [ 7121.888276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7121.894526] Lustre: Skipped 1 previous similar message [ 7127.009773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7127.022029] Lustre: Skipped 1 previous similar message [ 7130.079441] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7130.577937] Lustre: server umount lustre-MDT0000 complete [ 7132.131098] LustreError: 211491:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7132.156930] LustreError: 211491:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7134.896349] LustreError: 209185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787398868 with bad export cookie 1732227609947818077 [ 7134.901803] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7134.906982] LustreError: 209185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7135.344282] Lustre: server umount lustre-MDT0001 complete [ 7149.869749] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 7153.217436] Key type lgssc unregistered [ 7153.445726] LNet: 212890:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7153.457559] LNetError: 212890:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7154.474845] LNet: Removed LNI 192.168.202.105@tcp [ 7155.508430] Key type .llcrypt unregistered [ 7155.509911] Key type ._llcrypt unregistered [ 7176.510677] Key type ._llcrypt registered [ 7176.515675] Key type .llcrypt registered [ 7176.702992] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 7177.719424] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7177.794698] alg: No test for adler32 (adler32-zlib) [ 7178.886821] Lustre: Lustre: Build Version: 2.17.54_110_g2f98931 [ 7179.267978] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 7181.008448] Key type lgssc registered [ 7182.362677] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7190.109714] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7190.136445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7191.705389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7196.287323] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7201.253023] LustreError: 214159:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7201.291031] LustreError: 214159:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7205.769564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7206.234256] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7212.178499] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7221.484980] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7227.772688] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7235.248648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7235.556134] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7236.620771] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 7243.238108] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7254.346503] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7277.537069] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7277.543314] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7277.548853] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7277.566159] Lustre: Skipped 1 previous similar message [ 7277.586616] Lustre: Skipped 1 previous similar message [ 7279.104269] Lustre: server umount lustre-OST0000 complete [ 7287.777634] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7287.782407] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7287.794156] Lustre: Skipped 1 previous similar message [ 7287.827065] Lustre: Skipped 1 previous similar message [ 7290.069837] Lustre: server umount lustre-MDT0000 complete [ 7292.899571] LustreError: 214470:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7292.913683] LustreError: 214470:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7294.377547] LustreError: 216006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787399028 with bad export cookie 3670909876422212939 [ 7294.389809] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7294.402655] LustreError: 216006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7294.895154] Lustre: server umount lustre-MDT0001 complete [ 7308.182846] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 7310.852404] Key type lgssc unregistered [ 7311.092708] LNet: 217844:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7311.101607] LNetError: 217844:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7312.187211] LNet: Removed LNI 192.168.202.105@tcp [ 7313.061655] Key type .llcrypt unregistered [ 7313.063874] Key type ._llcrypt unregistered [ 7335.937186] Key type ._llcrypt registered [ 7335.939976] Key type .llcrypt registered [ 7336.038839] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 7337.326124] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7337.439922] alg: No test for adler32 (adler32-zlib) [ 7338.641486] Lustre: Lustre: Build Version: 2.17.54_110_g2f98931 [ 7338.971526] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 7340.655269] Key type lgssc registered [ 7341.837602] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7350.238200] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7350.260331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7351.721790] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7356.175367] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7362.038495] LustreError: 219112:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7362.061716] LustreError: 219112:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7366.335289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7366.956627] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7373.330986] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7383.638928] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7391.134296] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7399.078604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7399.683869] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7408.440790] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7417.864135] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7421.192119] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 7432.702396] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7432.839835] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7433.210913] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7440.016204] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7448.070340] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7450.135571] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7462.373136] Lustre: Permanently deactivating lustre-OST0001 [ 7462.415101] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7472.717926] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 7480.412283] Lustre: Permanently reactivating lustre-OST0001 [ 7480.433329] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7480.445348] Lustre: Skipped 2 previous similar messages [ 7481.843137] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7481.852842] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7481.858985] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7481.868732] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7481.893401] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7492.065991] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7492.066540] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7492.076230] LustreError: Skipped 1 previous similar message [ 7492.082933] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7492.094453] Lustre: Skipped 2 previous similar messages [ 7494.127758] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7494.139390] Lustre: Skipped 2 previous similar messages [ 7495.594404] Lustre: server umount lustre-OST0001 complete [ 7499.240671] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7499.240872] LustreError: 220935:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7499.243176] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7499.260899] Lustre: Skipped 1 previous similar message [ 7499.307389] LustreError: 220935:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7504.357574] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7504.359233] LustreError: 220932:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7504.361707] Lustre: Skipped 2 previous similar messages [ 7504.378492] LustreError: 220932:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7505.562296] Lustre: server umount lustre-OST0000 complete [ 7509.472575] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7509.490709] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7509.496553] Lustre: Skipped 1 previous similar message [ 7515.699090] Lustre: server umount lustre-MDT0000 complete [ 7519.715698] LustreError: 222426:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7519.749336] LustreError: 222426:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7519.812952] LustreError: 219094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787399253 with bad export cookie 14815155752197050644 [ 7519.822242] LustreError: 219094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7519.824270] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7520.336691] Lustre: server umount lustre-MDT0001 complete [ 7534.113686] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 7536.186700] Key type lgssc unregistered [ 7536.405921] LNet: 224182:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7536.411424] LNetError: 224182:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7536.423061] LNet: Removed LNI 192.168.202.105@tcp [ 7536.996588] Key type .llcrypt unregistered [ 7536.999795] Key type ._llcrypt unregistered [ 7581.930369] Key type ._llcrypt registered [ 7581.932798] Key type .llcrypt registered [ 7582.087706] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 7583.168486] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7583.439520] alg: No test for adler32 (adler32-zlib) [ 7584.705257] Lustre: Lustre: Build Version: 2.17.54_110_g2f98931 [ 7585.023440] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 7586.751737] Key type lgssc registered [ 7588.276313] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7596.471430] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7596.487406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7597.844487] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7597.862310] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7598.044917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7603.874282] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7612.432826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7612.495744] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7612.520182] Lustre: 226390:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7612.693470] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7617.348942] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7625.950705] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7631.975375] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7638.777745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7638.917510] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7639.141470] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7645.603032] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7648.845214] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 7652.692966] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7656.385604] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7660.655342] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7670.596969] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7680.760428] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7693.185213] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7693.191348] Lustre: Skipped 1 previous similar message [ 7715.373482] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7715.378598] Lustre: Skipped 2 previous similar messages [ 7752.897693] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7752.903763] Lustre: Skipped 4 previous similar messages [ 7827.958063] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7827.964715] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7827.968826] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7827.975761] Lustre: Skipped 1 previous similar message [ 7827.994394] Lustre: Skipped 1 previous similar message [ 7831.032639] Lustre: server umount lustre-OST0000 complete [ 7838.176474] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7838.182222] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7838.196037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7838.219245] LustreError: Skipped 1 previous similar message [ 7841.277691] Lustre: server umount lustre-MDT0000 complete [ 7845.069424] LustreError: 229229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787399578 with bad export cookie 9831288015304217402 [ 7845.072093] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7845.079184] LustreError: 229229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7845.392986] Lustre: server umount lustre-MDT0001 complete [ 7859.115500] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 7863.326801] Key type lgssc unregistered [ 7863.795573] LNet: 231179:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7863.802442] LNetError: 231179:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7863.816913] LNet: Removed LNI 192.168.202.105@tcp [ 7865.076777] Key type .llcrypt unregistered [ 7865.093693] Key type ._llcrypt unregistered [ 7894.920947] Key type ._llcrypt registered [ 7894.923449] Key type .llcrypt registered [ 7895.010825] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 7895.837061] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7895.971111] alg: No test for adler32 (adler32-zlib) [ 7897.000512] Lustre: Lustre: Build Version: 2.17.54_110_g2f98931 [ 7897.201864] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 7898.863195] Key type lgssc registered [ 7899.757839] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7906.840167] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7906.865802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7908.341093] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7913.405590] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7918.573020] LustreError: 232594:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7918.603977] LustreError: 232594:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7922.393704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7922.748136] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7928.152523] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7935.000235] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7940.326814] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7947.141109] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7947.567197] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7953.090262] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_default_debug -1 all [ 7953.723540] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7960.185560] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7978.475246] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7978.493778] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7978.494726] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7979.466105] Lustre: server umount lustre-OST0000 complete [ 7984.108472] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7984.123653] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7984.136795] Lustre: Skipped 1 previous similar message [ 7984.144276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7984.156724] Lustre: Skipped 1 previous similar message [ 7989.304473] Lustre: server umount lustre-MDT0000 complete [ 7989.729196] LustreError: 233323:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7989.754941] LustreError: 233323:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7993.334779] LustreError: 232574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787399727 with bad export cookie 15453098194136347254 [ 7993.342475] LustreError: MGC192.168.202.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7993.350188] LustreError: 232574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7993.880514] Lustre: server umount lustre-MDT0001 complete [ 8005.078386] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 8008.108278] Key type lgssc unregistered [ 8008.532463] LNet: 236289:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8008.546596] LNetError: 236289:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8008.571468] LNet: Removed LNI 192.168.202.105@tcp [ 8009.362798] Key type .llcrypt unregistered [ 8009.365968] Key type ._llcrypt unregistered [ 8034.840307] Key type ._llcrypt registered [ 8034.843911] Key type .llcrypt registered [ 8034.917651] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 8036.234496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8036.303194] alg: No test for adler32 (adler32-zlib) [ 8037.355405] Lustre: Lustre: Build Version: 2.17.54_110_g2f98931 [ 8037.589185] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 8039.264031] Key type lgssc registered [ 8040.500455] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8313.756886] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 8316.677929] Key type lgssc unregistered [ 8316.976305] LNet: 238312:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8316.987832] LNetError: 238312:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8317.027716] LNet: Removed LNI 192.168.202.105@tcp [ 8318.078490] Key type .llcrypt unregistered [ 8318.080282] Key type ._llcrypt unregistered [ 8348.331464] Key type ._llcrypt registered [ 8348.333872] Key type .llcrypt registered [ 8348.419499] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing set_hostid [ 8366.108400] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing load_modules_local [ 8367.429445] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8367.444044] alg: No test for adler32 (adler32-zlib) [ 8368.456544] Lustre: Lustre: Build Version: 2.17.54_110_g2f98931 [ 8368.688650] LNet: Added LNI 192.168.202.105@tcp [8/256/0/180] [ 8370.343760] Key type lgssc registered [ 8371.544246] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8403.513699] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 8115 sec ========== 08:02:15 (1787400135) [ 8405.417962] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 08:02:17 (1787400137) === [ 8409.111843] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 08:02:21 (1787400141) === [ 8452.647648] Lustre: DEBUG MARKER: oleg205-server.virtnet: executing unload_modules_local [ 8455.268572] Key type lgssc unregistered [ 8455.514687] LNet: 243558:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8455.520468] LNetError: 243558:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8455.556366] LNet: Removed LNI 192.168.202.105@tcp [ 8456.335337] Key type .llcrypt unregistered [ 8456.338150] Key type ._llcrypt unregistered