[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 470302397 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003384] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005008] kvm-guest: setup PV IPIs [ 0.007316] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010005] pid_max: default: 32768 minimum: 301 [ 0.011109] LSM: Security Framework initializing [ 0.012043] Yama: becoming mindful. [ 0.013043] SELinux: Initializing. [ 0.014080] *** VALIDATE selinux *** [ 0.022049] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026785] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027143] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028110] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029107] *** VALIDATE tmpfs *** [ 0.030466] *** VALIDATE proc *** [ 0.032151] *** VALIDATE cgroup *** [ 0.033012] *** VALIDATE cgroup2 *** [ 0.035106] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036170] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038040] Spectre V2 : User space: Vulnerable [ 0.039012] Speculative Store Bypass: Vulnerable [ 0.042000] debug: unmapping init [mem 0xffffffff94259000-0xffffffff94260fff] [ 0.043619] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044885] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045022] ... version: 2 [ 0.046015] ... bit width: 48 [ 0.047013] ... generic registers: 4 [ 0.048077] ... value mask: 0000ffffffffffff [ 0.049014] ... max period: 00007fffffffffff [ 0.050016] ... fixed-purpose events: 3 [ 0.051012] ... event mask: 000000070000000f [ 0.052310] rcu: Hierarchical SRCU implementation. [ 0.054935] smp: Bringing up secondary CPUs ... [ 0.056032] x86: Booting SMP configuration: [ 0.057013] .... node #0, CPUs: #1 #2 #3 [ 0.065017] smp: Brought up 1 node, 4 CPUs [ 0.067012] smpboot: Max logical packages: 1 [ 0.068016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.130717] node 0 deferred pages initialised in 61ms [ 0.139168] devtmpfs: initialized [ 0.143459] x86/mm: Memory block size: 128MB [ 0.151677] gcov: version magic: 0x41383552 [ 0.155307] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.156076] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.157506] pinctrl core: initialized pinctrl subsystem [ 0.158161] [ 0.158602] ************************************************************* [ 0.159012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.160013] ** ** [ 0.161064] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.162011] ** ** [ 0.163013] ** This means that this kernel is built to expose internal ** [ 0.164013] ** IOMMU data structures, which may compromise security on ** [ 0.165012] ** your system. ** [ 0.166013] ** ** [ 0.167016] ** If you see this message and you are not debugging the ** [ 0.168018] ** kernel, report this immediately to your vendor! ** [ 0.169013] ** ** [ 0.170015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.171013] ************************************************************* [ 0.172784] NET: Registered protocol family 16 [ 0.175365] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.182028] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.184060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.189095] cpuidle: using governor menu [ 0.193010] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.198025] PCI: Using configuration type 1 for base access [ 0.199249] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.210325] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.215021] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.217202] cryptd: max_cpu_qlen set to 1000 [ 0.224380] ACPI: Added _OSI(Module Device) [ 0.226017] ACPI: Added _OSI(Processor Device) [ 0.228014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.232015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.243156] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.249357] ACPI: Interpreter enabled [ 0.250039] ACPI: PM: (supports S0 S3 S4 S5) [ 0.251008] ACPI: Using IOAPIC for interrupt routing [ 0.253075] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.257555] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.266818] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.270133] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.274024] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.278081] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.287054] acpiphp: Slot [2] registered [ 0.289119] acpiphp: Slot [5] registered [ 0.291161] acpiphp: Slot [6] registered [ 0.293211] acpiphp: Slot [7] registered [ 0.297581] acpiphp: Slot [8] registered [ 0.300189] acpiphp: Slot [9] registered [ 0.303382] acpiphp: Slot [10] registered [ 0.306170] acpiphp: Slot [3] registered [ 0.308132] acpiphp: Slot [4] registered [ 0.310112] acpiphp: Slot [11] registered [ 0.312167] acpiphp: Slot [12] registered [ 0.314110] acpiphp: Slot [13] registered [ 0.316474] acpiphp: Slot [14] registered [ 0.319117] acpiphp: Slot [15] registered [ 0.321118] acpiphp: Slot [16] registered [ 0.324122] acpiphp: Slot [17] registered [ 0.327273] acpiphp: Slot [18] registered [ 0.329116] acpiphp: Slot [19] registered [ 0.331129] acpiphp: Slot [20] registered [ 0.334087] acpiphp: Slot [21] registered [ 0.338127] acpiphp: Slot [22] registered [ 0.339111] acpiphp: Slot [23] registered [ 0.340106] acpiphp: Slot [24] registered [ 0.343103] acpiphp: Slot [25] registered [ 0.346365] acpiphp: Slot [26] registered [ 0.348099] acpiphp: Slot [27] registered [ 0.350172] acpiphp: Slot [28] registered [ 0.352098] acpiphp: Slot [29] registered [ 0.354103] acpiphp: Slot [30] registered [ 0.355076] acpiphp: Slot [31] registered [ 0.356123] PCI host bridge to bus 0000:00 [ 0.357022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.359024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.361022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.362022] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.364031] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.367053] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.369174] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.373131] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.374000] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.389888] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.395980] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.399022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.400000] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.401016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.406123] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.414053] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.419061] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.429106] pci 0000:00:01.3: quirk_piix4_acpi+0x0/0x1e0 took 14648 usecs [ 0.437201] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.446017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.474015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.489015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.496214] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.508015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.516000] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.546015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.564014] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.579016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.603018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.640017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.652528] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.662014] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.672012] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.705019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.726000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.733014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.751000] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.796017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.823799] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.843020] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.860016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.891015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.907438] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.920014] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.931015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.963017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.992000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.995772] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 1.000639] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 1.004423] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 1.007221] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 1.015038] iommu: Default domain type: Passthrough [ 1.017694] SCSI subsystem initialized [ 1.019145] ACPI: bus type USB registered [ 1.023132] usbcore: registered new interface driver usbfs [ 1.030070] usbcore: registered new interface driver hub [ 1.035217] usbcore: registered new device driver usb [ 1.036179] pps_core: LinuxPPS API ver. 1 registered [ 1.039016] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.045056] PTP clock support registered [ 1.052121] EDAC MC: Ver: 3.0.0 [ 1.054335] PCI: Using ACPI for IRQ routing [ 1.055560] NetLabel: Initializing [ 1.058013] NetLabel: domain hash size = 128 [ 1.062012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.064078] NetLabel: unlabeled traffic allowed by default [ 1.071098] vgaarb: loaded [ 1.077093] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.080012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.089531] clocksource: Switched to clocksource kvm-clock [ 1.284605] VFS: Disk quotas dquot_6.6.0 [ 1.286788] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.290076] *** VALIDATE ramfs *** [ 1.293674] *** VALIDATE hugetlbfs *** [ 1.295926] pnp: PnP ACPI init [ 1.299492] pnp: PnP ACPI: found 6 devices [ 1.331346] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.339051] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.341482] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.344351] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.347761] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.350607] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.353281] NET: Registered protocol family 2 [ 1.362219] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.368177] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.371305] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.383749] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.387180] TCP: Hash tables configured (established 65536 bind 65536) [ 1.392685] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.399949] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.405687] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.409429] NET: Registered protocol family 1 [ 1.415962] RPC: Registered named UNIX socket transport module. [ 1.419044] RPC: Registered udp transport module. [ 1.422608] RPC: Registered tcp transport module. [ 1.425655] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.428734] NET: Registered protocol family 44 [ 1.431895] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.435113] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.439359] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.443209] PCI: CLS 0 bytes, default 64 [ 1.445159] Unpacking initramfs... [ 4.450306] debug: unmapping init [mem 0xffff94273cc54000-0xffff94273ffbffff] [ 4.455200] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 4.458667] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 4.462240] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 5.093997] Initialise system trusted keyrings [ 5.095536] Key type blacklist registered [ 5.098780] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 5.121128] zbud: loaded [ 5.127611] *** VALIDATE nfs *** [ 5.128631] *** VALIDATE nfs4 *** [ 5.130034] pstore: using deflate compression [ 5.133548] Platform Keyring initialized [ 5.261497] NET: Registered protocol family 38 [ 5.264662] Key type asymmetric registered [ 5.266328] Asymmetric key parser 'x509' registered [ 5.268761] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 5.273056] io scheduler mq-deadline registered [ 5.275448] io scheduler kyber registered [ 5.277445] io scheduler bfq registered [ 5.280163] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 5.284365] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 5.288416] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 5.291909] ACPI: Power Button [PWRF] [ 5.298848] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 5.308714] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 5.345580] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 5.363974] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 5.394863] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 5.438892] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 5.467084] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 5.473621] Non-volatile memory driver v1.3 [ 5.476914] Linux agpgart interface v0.103 [ 5.523819] virtio_blk virtio1: [vda] 149816 512-byte logical blocks (76.7 MB/73.2 MiB) [ 5.530881] vda: detected capacity change from 0 to 76705792 [ 5.566110] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 5.570917] vdb: detected capacity change from 0 to 1073741824 [ 5.607310] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.612988] vdc: detected capacity change from 0 to 2621440000 [ 5.657375] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.664759] vdd: detected capacity change from 0 to 2621440000 [ 5.699492] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.703474] vde: detected capacity change from 0 to 4294967296 [ 5.744718] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.750682] vdf: detected capacity change from 0 to 4294967296 [ 5.768354] libphy: Fixed MDIO Bus: probed [ 5.776216] usbcore: registered new interface driver usbserial_generic [ 5.779201] usbserial: USB Serial support registered for generic [ 5.780895] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.785873] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.787759] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.790870] mousedev: PS/2 mouse device common for all mice [ 5.797128] rtc_cmos 00:05: RTC can wake from S4 [ 5.807113] rtc_cmos 00:05: registered as rtc0 [ 5.810600] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.814816] intel_pstate: CPU model not supported [ 5.819179] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.819320] hid: raw HID events driver (C) Jiri Kosina [ 5.829411] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.832294] usbcore: registered new interface driver usbhid [ 5.832304] usbhid: USB HID core driver [ 5.832417] drop_monitor: Initializing network drop monitor service [ 5.832547] Initializing XFRM netlink socket [ 5.832909] NET: Registered protocol family 10 [ 5.836474] Segment Routing with IPv6 [ 5.843351] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.844105] NET: Registered protocol family 17 [ 5.844987] mpls_gso: MPLS GSO support [ 5.874929] RAS: Correctable Errors collector initialized. [ 5.878217] AVX version of gcm_enc/dec engaged. [ 5.882217] AES CTR mode by8 optimization enabled [ 6.134596] sched_clock: Marking stable (6134567370, 0)->(7220775985, -1086208615) [ 6.146378] registered taskstats version 1 [ 6.150503] Loading compiled-in X.509 certificates [ 6.159664] zswap: loaded using pool lzo/zbud [ 6.195966] Key type big_key registered [ 6.213285] Key type encrypted registered [ 6.214958] ima: No TPM chip found, activating TPM-bypass! [ 6.217327] ima: Allocated hash algorithm: sha1 [ 6.219976] ima: No architecture policies found [ 6.224375] evm: Initialising EVM extended attributes: [ 6.228351] evm: security.selinux [ 6.231294] evm: security.ima [ 6.236715] evm: security.capability [ 6.238685] evm: HMAC attrs: 0x1 [ 6.252402] rtc_cmos 00:05: setting system clock to 2026-09-05 07:33:10 UTC (1788593590) [ 6.265596] debug: unmapping init [mem 0xffffffff95203000-0xffffffff953fffff] [ 6.272542] debug: unmapping init [mem 0xffffffff93f82000-0xffffffff94258fff] [ 6.286203] Write protecting the kernel read-only data: 28672k [ 6.293963] debug: unmapping init [mem 0xffffffff92603000-0xffffffff927fffff] [ 6.298708] debug: unmapping init [mem 0xffffffff92f14000-0xffffffff92ffffff] [ 6.366856] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 6.377262] systemd[1]: Detected virtualization kvm. [ 6.379096] systemd[1]: Detected architecture x86-64. [ 6.381539] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 6.412588] systemd[1]: No hostname configured. [ 6.414114] systemd[1]: Set hostname to . [ 6.416837] random: systemd: uninitialized urandom read (16 bytes read) [ 6.419725] systemd[1]: Initializing machine ID from random generator. [ 6.493789] random: ln: uninitialized urandom read (6 bytes read) [ 6.624159] random: systemd: uninitialized urandom read (16 bytes read) [ 6.632696] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 6.654426] systemd[1]: Starting Apply Kernel Variables... Starting Apply Kernel Variables... [ 6.665130] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.909565] device-mapper: uevent: version 1.0.3 [ 7.914923] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ 9.643737] virtio_net virtio0 ens2: renamed from eth0 Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 9.779505] scsi host0: ata_piix [ 9.826684] scsi host1: ata_piix [ 9.842124] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 9.844267] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.877035] random: fast init done [ 14.780683] random: crng init done [ 14.782806] random: 7 urandom warning(s) missed due to ratelimiting [ 14.958524] dracut-initqueue[594]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 17.326928] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 20.061448] printk: systemd: 25 output lines suppressed due to ratelimiting [ 20.828417] SELinux: Disabled at runtime. [ 21.058327] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 21.089878] systemd[1]: Detected virtualization kvm. [ 21.093162] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 23.193364] systemd[1]: initrd-switch-root.service: Succeeded. [ 23.206964] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 23.236190] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 23.251578] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 23.263080] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 23.277922] systemd[1]: Starting Journal Service... Starting Journal Service... [ 23.296639] systemd[1]: Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice User and Session Slice. [ OK ] Listening on udev Control Socket. Starting Remount Root and Kernel File Systems... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ 23.476206] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting POSIX Message Queue File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on RPCbind Server Activation Socket. Mounting Huge Pages File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-getty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target RPC Port Mapper. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Kernel Socket. Starting Apply Kernel Variables... [ OK ] Reached target Slices. Mounting Kernel Debug File System... Starting udev Coldplug all Devices... [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 25.394440] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 26.211390] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 26.345783] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 26.862914] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 26.933079] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit)[ 32.162872] Key type dns_resolver registered [*** ] A start job is running for Configur…-only root support (9s / no limit)[ 32.739577] NFS: Registering the id_resolver key type [ 32.744979] Key type id_resolver registered [ 32.748247] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... Starting Login Service... Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg628-server login: [ 70.278241] hrtimer: interrupt took 2376291 ns [ 87.966521] libcfs: loading out-of-tree module taints kernel. [ 88.022740] Key type ._llcrypt registered [ 88.025432] Key type .llcrypt registered [ 88.176777] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_hostid [ 108.423870] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 110.154106] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 110.190591] alg: No test for adler32 (adler32-zlib) [ 111.610517] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 112.660508] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 114.456146] Key type lgssc registered [ 116.263788] Lustre: Echo OBD driver; http://www.lustre.org/ [ 134.492185] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 172.435559] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 185.091255] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 185.114113] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 186.412118] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 186.458691] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 186.589440] Lustre: lustre-MDT0000: new disk, initializing [ 186.708086] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 186.731638] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 191.490904] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 205.427666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 205.487264] Lustre: 6525:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 205.540809] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 205.548302] Lustre: Skipped 1 previous similar message [ 205.648190] Lustre: lustre-MDT0001: new disk, initializing [ 205.744917] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 205.779399] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 205.794861] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 210.398844] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 216.360290] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 226.091081] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 226.379846] Lustre: lustre-OST0000: new disk, initializing [ 226.383213] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 226.388366] Lustre: 8465:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 226.445086] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 231.883224] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 235.049972] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 235.061745] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 235.145441] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 245.158545] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 245.264195] Lustre: lustre-OST0001: new disk, initializing [ 245.267431] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 245.274989] Lustre: 9537:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 245.319075] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 251.147800] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 252.468522] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 252.480077] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 252.525855] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 262.655713] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 273.551227] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 279.840472] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing check_logdir /tmp/testlogs/ [ 284.401619] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing yml_node [ 288.375322] Lustre: DEBUG MARKER: Client: 2.17.58.2 [ 290.736161] Lustre: DEBUG MARKER: MDS: 2.17.58.2 [ 293.279966] Lustre: DEBUG MARKER: OSS: 2.17.58.2 [ 294.792380] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Sep 5 03:37:58 EDT 2026 [ 311.261822] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 312.785591] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 329.185698] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 329.198856] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 329.201671] Lustre: Skipped 1 previous similar message [ 329.218448] Lustre: Skipped 3 previous similar messages [ 334.306253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 334.315679] Lustre: Skipped 3 previous similar messages [ 335.408147] Lustre: server umount lustre-MDT0000 complete [ 338.264605] LustreError: 10355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788593922 with bad export cookie 1790458052055670430 [ 338.271924] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 338.285204] LustreError: 10355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 339.434514] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 339.454294] Lustre: Skipped 2 previous similar messages [ 339.464700] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 339.472139] Lustre: Skipped 1 previous similar message [ 343.466055] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 343.477091] Lustre: Skipped 1 previous similar message [ 344.859139] Lustre: server umount lustre-MDT0001 complete [ 348.813703] Lustre: server umount lustre-OST0000 complete [ 352.171394] Lustre: server umount lustre-OST0001 complete [ 360.234786] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_hostid [ 369.754572] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 404.648343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 404.901719] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 404.969450] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 405.086596] Lustre: lustre-MDT0000: new disk, initializing [ 405.174158] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 405.200321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 409.912649] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 420.094985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 420.203686] Lustre: 15090:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 420.284489] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 420.295816] Lustre: Skipped 1 previous similar message [ 420.414238] Lustre: lustre-MDT0001: new disk, initializing [ 420.523348] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 420.557683] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 420.571845] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 424.780562] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 433.382850] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 438.243581] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 444.354989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 444.684593] Lustre: lustre-OST0000: new disk, initializing [ 444.688213] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 444.696071] Lustre: 17061:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 444.760200] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 445.821716] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 445.838771] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 445.955350] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 450.639619] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 459.972969] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 465.494343] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 465.721307] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 470.846163] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 471.018736] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 476.650494] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 476.654479] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 476.663578] LustreError: Skipped 1 previous similar message [ 476.668912] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 476.690536] Lustre: Skipped 2 previous similar messages [ 486.882875] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 486.883754] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 4. Is it stuck? [ 486.896576] Lustre: Skipped 4 previous similar messages [ 487.141930] Lustre: server umount lustre-OST0000 complete [ 492.003888] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 496.575933] Lustre: server umount lustre-MDT0000 complete [ 497.124074] LustreError: 15097:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 497.144724] LustreError: 15097:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 500.270713] LustreError: 15082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788594084 with bad export cookie 1790458052055671963 [ 500.273278] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 500.279871] LustreError: 15082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 500.665392] Lustre: server umount lustre-MDT0001 complete [ 503.908932] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 03:41:27 (1788594087) [ 509.892044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 510.344055] LustreError: 19326:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 510.494095] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 515.156656] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 515.563548] LustreError: 19327:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 520.678326] LustreError: 19326:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 523.293959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 528.012732] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 534.783661] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 539.668604] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 545.650888] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 551.348571] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 558.215237] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 565.732038] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 565.738502] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 565.744760] Lustre: Skipped 1 previous similar message [ 565.748263] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 565.765101] Lustre: Skipped 2 previous similar messages [ 570.862908] Lustre: server umount lustre-OST0000 complete [ 574.959942] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 574.978517] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 575.006434] Lustre: Skipped 1 previous similar message [ 580.753850] Lustre: server umount lustre-MDT0000 complete [ 581.093583] LustreError: 20053:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 583.679723] LustreError: 19306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788594167 with bad export cookie 1790458052055672838 [ 583.683680] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 583.687065] LustreError: 19306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 584.066932] Lustre: server umount lustre-MDT0001 complete [ 593.588904] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 596.063552] Key type lgssc unregistered [ 596.334596] LNet: 22855:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 596.340991] LNetError: 22855:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 596.359695] LNet: Removed LNI 192.168.206.128@tcp [ 597.342135] Key type .llcrypt unregistered [ 597.345113] Key type ._llcrypt unregistered [ 617.505441] Key type ._llcrypt registered [ 617.509701] Key type .llcrypt registered [ 617.655611] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 618.652320] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 618.789830] alg: No test for adler32 (adler32-zlib) [ 619.955653] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 620.217532] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 621.880217] Key type lgssc registered [ 623.071610] Lustre: Echo OBD driver; http://www.lustre.org/ [ 631.028364] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 631.058519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 632.508411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 635.653205] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 642.020194] LustreError: 24244:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 642.051396] LustreError: 24244:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 643.482396] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 643.930768] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 649.236626] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 656.016074] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 662.811505] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 670.199645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 670.607622] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 676.352736] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 676.782183] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 683.952930] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 699.367191] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 699.369821] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 699.380928] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 699.392992] Lustre: Skipped 1 previous similar message [ 699.404782] Lustre: Skipped 1 previous similar message [ 701.398446] Lustre: server umount lustre-OST0000 complete [ 705.505705] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 705.512781] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 705.525863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 709.601043] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 709.610179] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 711.487795] Lustre: server umount lustre-MDT0000 complete [ 714.722316] LustreError: 24970:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 714.746398] LustreError: 24970:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 715.122540] LustreError: 26087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788594299 with bad export cookie 12246412099231635223 [ 715.127844] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 715.133376] LustreError: 26087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 715.541572] Lustre: server umount lustre-MDT0001 complete [ 725.677983] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 728.094693] Key type lgssc unregistered [ 728.392683] LNet: 28219:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 728.400596] LNetError: 28219:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 729.447440] LNet: Removed LNI 192.168.206.128@tcp [ 730.257268] Key type .llcrypt unregistered [ 730.259884] Key type ._llcrypt unregistered [ 751.687775] Key type ._llcrypt registered [ 751.690566] Key type .llcrypt registered [ 751.782363] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 753.098197] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 753.114900] alg: No test for adler32 (adler32-zlib) [ 754.217534] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 754.527385] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 756.192220] Key type lgssc registered [ 757.399102] Lustre: Echo OBD driver; http://www.lustre.org/ [ 765.475834] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 765.502166] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 766.959859] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 771.367224] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 777.187929] LustreError: 29623:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 777.207508] LustreError: 29623:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 778.682632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 779.121660] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 783.509975] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 790.381841] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 795.018503] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 807.799277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 808.199366] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 812.299825] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 814.165025] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 821.898986] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 832.993813] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 833.000944] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 833.017042] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 835.042311] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 835.049142] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 835.049494] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 838.596453] Lustre: server umount lustre-OST0000 complete [ 845.281992] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 845.290042] Lustre: Skipped 1 previous similar message [ 845.290898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 845.313000] Lustre: Skipped 2 previous similar messages [ 848.043315] Lustre: server umount lustre-MDT0000 complete [ 850.404130] LustreError: 29618:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 850.420355] LustreError: 29618:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 851.421489] LustreError: 31809:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788594435 with bad export cookie 3462126877285681412 [ 851.428883] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 851.435350] LustreError: 31809:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 851.678115] Lustre: server umount lustre-MDT0001 complete [ 862.205509] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 865.600826] Key type lgssc unregistered [ 866.073967] LNet: 33500:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 866.100838] LNetError: 33500:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 866.144850] LNet: Removed LNI 192.168.206.128@tcp [ 867.280498] Key type .llcrypt unregistered [ 867.282315] Key type ._llcrypt unregistered [ 889.845838] Key type ._llcrypt registered [ 889.848264] Key type .llcrypt registered [ 889.977791] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 891.073858] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 891.147732] alg: No test for adler32 (adler32-zlib) [ 892.244774] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 892.466213] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 894.208348] Key type lgssc registered [ 895.505154] Lustre: Echo OBD driver; http://www.lustre.org/ [ 903.376637] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 903.420260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 904.944683] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 909.008382] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 915.430104] LustreError: 34911:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 915.451537] LustreError: 34911:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 916.527913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 916.994324] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 921.227691] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 927.620660] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 932.163899] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 938.270391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 938.775668] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 945.375708] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 950.073622] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 952.704353] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 962.529584] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 962.534354] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 962.543152] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 963.053655] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 963.060491] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 963.072621] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 965.610786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 965.619380] Lustre: Skipped 1 previous similar message [ 967.149303] Lustre: server umount lustre-OST0000 complete [ 970.724963] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 970.731718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 970.738845] Lustre: Skipped 1 previous similar message [ 970.758729] Lustre: Skipped 1 previous similar message [ 975.844640] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 975.854080] Lustre: Skipped 1 previous similar message [ 976.828898] Lustre: server umount lustre-MDT0000 complete [ 980.633595] LustreError: 34892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788594564 with bad export cookie 3242230930725110554 [ 980.657643] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 980.989213] Lustre: server umount lustre-MDT0001 complete [ 992.803872] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 995.932246] Key type lgssc unregistered [ 996.243130] LNet: 38439:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 996.255900] LNetError: 38439:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 996.277213] LNet: Removed LNI 192.168.206.128@tcp [ 997.105889] Key type .llcrypt unregistered [ 997.107750] Key type ._llcrypt unregistered [ 1020.449389] Key type ._llcrypt registered [ 1020.451653] Key type .llcrypt registered [ 1020.569938] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 1021.628311] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1021.736432] alg: No test for adler32 (adler32-zlib) [ 1022.828511] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 1023.094942] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 1024.760171] Key type lgssc registered [ 1025.850947] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1031.981547] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1031.995055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1033.302072] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1036.086498] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 1042.599804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1042.990147] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1048.386633] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 1055.953818] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1062.053498] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1068.026652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1068.275169] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1073.061793] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 1078.765887] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1079.547650] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1084.897322] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1084.903640] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1084.907314] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1084.921660] Lustre: Skipped 1 previous similar message [ 1084.935496] Lustre: Skipped 1 previous similar message [ 1086.499418] Lustre: lustre-OST0000: Not available for connect from 192.168.206.28@tcp (stopping) [ 1090.034035] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1090.038961] Lustre: Skipped 1 previous similar message [ 1095.146814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1095.159586] Lustre: Skipped 2 previous similar messages [ 1099.233124] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1099.385043] Lustre: server umount lustre-OST0000 complete [ 1119.717638] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1119.725758] LustreError: Skipped 1 previous similar message [ 1119.728636] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1119.746566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1119.752386] Lustre: Skipped 1 previous similar message [ 1120.744574] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1124.303602] Lustre: server umount lustre-MDT0000 complete [ 1124.846756] LustreError: 39847:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1124.871740] LustreError: 39847:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1127.747805] LustreError: 39831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788594711 with bad export cookie 4735398993141185891 [ 1127.759442] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1127.762457] LustreError: 39831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1142.240207] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1142.738861] Lustre: server umount lustre-MDT0001 complete [ 1154.072927] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 1156.324914] Key type lgssc unregistered [ 1156.617733] LNet: 43571:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1156.624470] LNetError: 43571:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1156.639049] LNet: Removed LNI 192.168.206.128@tcp [ 1157.361136] Key type .llcrypt unregistered [ 1157.364055] Key type ._llcrypt unregistered [ 1179.083375] Key type ._llcrypt registered [ 1179.085277] Key type .llcrypt registered [ 1179.239933] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 1180.428727] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1180.482495] alg: No test for adler32 (adler32-zlib) [ 1181.710553] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 1182.099523] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 1183.840162] Key type lgssc registered [ 1185.132618] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1192.163627] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1192.193761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1193.667028] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1198.216643] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 1203.174211] LustreError: 44980:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1203.196710] LustreError: 44980:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1205.724684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1206.097303] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1210.654627] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 1217.562970] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1223.038613] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1229.387970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1229.775632] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1235.412763] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 1238.036501] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1242.815267] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1250.852234] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.28@tcp (stopping) [ 1251.360089] Lustre: server umount lustre-MDT0000 complete [ 1252.323038] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1252.340221] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1252.357411] LustreError: 44980:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1253.349950] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1253.361145] LustreError: 44977:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1253.376807] LustreError: 44977:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1254.626876] LustreError: 44960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788594838 with bad export cookie 2344026045990942596 [ 1254.641536] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1255.949011] LustreError: 44976:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1255.994374] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.28@tcp (stopping) [ 1258.470109] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1258.493279] Lustre: Skipped 1 previous similar message [ 1258.501838] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1261.069973] LustreError: 44977:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1261.102604] LustreError: 44977:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1261.212806] Lustre: server umount lustre-MDT0001 complete [ 1281.858200] Lustre: server umount lustre-OST0000 complete [ 1297.216148] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 1300.781974] Key type lgssc unregistered [ 1301.270408] LNet: 48834:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1301.290569] LNetError: 48834:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1301.321255] LNet: Removed LNI 192.168.206.128@tcp [ 1302.542418] Key type .llcrypt unregistered [ 1302.547090] Key type ._llcrypt unregistered [ 1327.104937] Key type ._llcrypt registered [ 1327.110932] Key type .llcrypt registered [ 1327.183470] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 1328.014207] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1328.058767] alg: No test for adler32 (adler32-zlib) [ 1329.169191] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 1329.502941] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 1331.224154] Key type lgssc registered [ 1332.321282] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1339.368596] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1339.386930] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1354.976368] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1360.096598] LustreError: 50216:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.128@tcp: failed processing log, type 4: rc = -110 [ 1386.784855] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1393.457776] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 1401.071618] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1566.040557] Lustre: server umount lustre-OST0000 complete [ 1580.024028] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 1583.085816] Key type lgssc unregistered [ 1583.407478] LNet: 51891:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1583.422850] LNetError: 51891:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1583.448273] LNet: Removed LNI 192.168.206.128@tcp [ 1584.326792] Key type .llcrypt unregistered [ 1584.331974] Key type ._llcrypt unregistered [ 1605.771842] Key type ._llcrypt registered [ 1605.776699] Key type .llcrypt registered [ 1605.891439] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 1606.921285] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1606.972456] alg: No test for adler32 (adler32-zlib) [ 1608.236129] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 1608.573890] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 1610.232236] Key type lgssc registered [ 1611.376791] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1618.726565] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1618.757821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1620.295538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1624.220439] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 1630.692317] LustreError: 53309:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1630.719995] LustreError: 53309:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1632.683588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1632.991873] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1637.733068] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 1643.830794] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1649.197818] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1656.251525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1656.692607] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1662.745283] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 1664.895126] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1669.839304] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1674.209918] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1674.218783] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1674.236787] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1675.245199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1675.260651] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1675.279587] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1678.045379] Lustre: server umount lustre-OST0000 complete [ 1684.450112] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1684.458243] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1684.478267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1684.484599] Lustre: Skipped 1 previous similar message [ 1687.881300] Lustre: server umount lustre-MDT0000 complete [ 1690.595516] LustreError: 55151:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1690.616804] LustreError: 55151:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1691.120537] LustreError: 53288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788595275 with bad export cookie 17682011333116172104 [ 1691.129278] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1691.387396] Lustre: server umount lustre-MDT0001 complete [ 1701.163978] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 1703.871493] Key type lgssc unregistered [ 1704.146514] LNet: 56851:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1704.155985] LNetError: 56851:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1704.172398] LNet: Removed LNI 192.168.206.128@tcp [ 1704.965756] Key type .llcrypt unregistered [ 1704.967593] Key type ._llcrypt unregistered [ 1727.304934] Key type ._llcrypt registered [ 1727.313266] Key type .llcrypt registered [ 1727.401287] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 1728.323453] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1728.357802] alg: No test for adler32 (adler32-zlib) [ 1729.514849] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 1729.899818] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 1731.600657] Key type lgssc registered [ 1733.232982] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1741.213598] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1741.236715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1756.832416] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1761.952288] LustreError: 58235:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.128@tcp: failed processing log, type 4: rc = -110 [ 1788.577255] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1794.572835] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 1801.421770] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1807.545517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1808.163701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1808.253625] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1812.522663] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 1813.478143] LustreError: 58261:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1813.507362] LustreError: 58261:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1818.597349] LustreError: 58260:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1820.495184] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1820.860215] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1825.193722] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 1831.784761] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1837.893686] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1841.636230] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1841.645705] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1841.662884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1843.699364] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1843.707212] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1843.731425] Lustre: Skipped 1 previous similar message [ 1846.252482] Lustre: server umount lustre-OST0000 complete [ 1869.282760] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1869.289214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1869.292816] Lustre: Skipped 1 previous similar message [ 1869.312391] Lustre: Skipped 1 previous similar message [ 1873.666207] Lustre: server umount lustre-MDT0000 complete [ 1875.427325] LustreError: 59277:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1875.448470] LustreError: 59277:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1877.018257] LustreError: 58241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788595461 with bad export cookie 13322914855994754492 [ 1877.020305] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1877.026659] LustreError: 58241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1883.484517] Lustre: server umount lustre-MDT0001 complete [ 1894.563877] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 1897.099047] Key type lgssc unregistered [ 1897.386847] LNet: 61976:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1897.399077] LNetError: 61976:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1897.423489] LNet: Removed LNI 192.168.206.128@tcp [ 1898.295381] Key type .llcrypt unregistered [ 1898.297314] Key type ._llcrypt unregistered [ 1919.519934] Key type ._llcrypt registered [ 1919.521882] Key type .llcrypt registered [ 1919.609263] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 1920.702077] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1920.745408] alg: No test for adler32 (adler32-zlib) [ 1921.823398] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 1922.076447] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 1923.745117] Key type lgssc registered [ 1924.700784] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1931.078323] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1931.115626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1932.531782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1936.370768] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 1941.987117] LustreError: 63383:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1942.007763] LustreError: 63383:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1943.769135] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1944.101326] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1948.214128] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 1955.031789] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1960.302405] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1966.660829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1966.990195] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1973.226764] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 1977.266603] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1980.196268] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1997.795909] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1997.818106] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1997.824903] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1997.834643] Lustre: Skipped 1 previous similar message [ 1997.848982] Lustre: Skipped 1 previous similar message [ 1998.177369] Lustre: server umount lustre-OST0000 complete [ 2002.915149] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2002.928130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2002.938520] Lustre: Skipped 1 previous similar message [ 2008.037691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2008.047614] Lustre: Skipped 1 previous similar message [ 2013.153847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2013.161031] Lustre: Skipped 1 previous similar message [ 2016.224082] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2016.489548] Lustre: server umount lustre-MDT0000 complete [ 2018.278209] LustreError: 63382:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2018.299323] LustreError: 63382:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2020.579887] LustreError: 63365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788595604 with bad export cookie 10247876536016437948 [ 2020.584001] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2020.594275] LustreError: 63365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2021.116271] Lustre: server umount lustre-MDT0001 complete [ 2033.650877] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 2036.765695] Key type lgssc unregistered [ 2037.132076] LNet: 66940:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2037.143392] LNetError: 66940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2037.162700] LNet: Removed LNI 192.168.206.128@tcp [ 2038.064665] Key type .llcrypt unregistered [ 2038.067092] Key type ._llcrypt unregistered [ 2064.796565] Key type ._llcrypt registered [ 2064.798609] Key type .llcrypt registered [ 2064.975948] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 2066.218687] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2066.257777] alg: No test for adler32 (adler32-zlib) [ 2067.472251] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 2067.821045] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 2069.537333] Key type lgssc registered [ 2070.765711] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2080.192921] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 04:07:44 (1788595664) [ 2087.225433] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2087.242784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2088.639519] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2093.522802] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 2098.151609] LustreError: 68641:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2098.166336] LustreError: 68641:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2102.134705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2102.576186] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2107.477319] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 2115.728380] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2121.340629] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2128.032779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2128.376743] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2133.478579] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2133.793700] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 2139.977114] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2149.421793] Lustre: Failing over lustre-MDT0000 [ 2149.939310] Lustre: server umount lustre-MDT0000 complete [ 2151.930089] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2151.943289] LustreError: 68637:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2151.946178] Lustre: Skipped 2 previous similar messages [ 2151.973536] LustreError: 68637:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2153.019968] LustreError: 68637:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2157.029043] LustreError: 68636:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2157.042558] LustreError: 68636:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2159.697600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2159.840959] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2160.075582] Lustre: *** cfs_fail_loc=135, val=0*** [ 2160.134910] LustreError: 71729:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2160.148123] LustreError: 71729:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.128@tcp: cfg command failed: rc = -2 [ 2160.170600] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2160.170600] [ 2160.176610] LustreError: MGC192.168.206.128@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2160.184146] LustreError: 71717:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2160.190126] LustreError: 71717:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [ 2160.194824] LustreError: 71717:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 2160.199326] LustreError: 71717:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2160.224505] Lustre: server umount lustre-MDT0000 complete [ 2160.230733] LustreError: 71717:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2162.154018] LustreError: 68641:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2162.170942] LustreError: 68641:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2166.571158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2166.734960] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2166.756839] LustreError: 72074:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGC192.168.206.128@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (00000000c415c4e6) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 2166.784057] LustreError: 68632:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2167.092018] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2167.138755] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2168.337801] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2171.087632] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 2172.409340] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2172.442880] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2172.482865] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2177.507459] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2177.524057] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2177.527160] Lustre: Skipped 1 previous similar message [ 2179.277895] Lustre: server umount lustre-OST0000 complete [ 2187.747397] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2187.757194] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2187.765852] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2187.765859] Lustre: Skipped 1 previous similar message [ 2187.799444] Lustre: Skipped 1 previous similar message [ 2189.255544] Lustre: server umount lustre-MDT0000 complete [ 2192.867563] LustreError: 68636:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2192.890644] LustreError: 68636:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2193.233978] LustreError: 71220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788595777 with bad export cookie 5294959005319148416 [ 2193.248201] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2193.249612] LustreError: 71220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2193.561782] Lustre: server umount lustre-MDT0001 complete [ 2205.118337] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 2208.662968] Key type lgssc unregistered [ 2209.001905] LNet: 73506:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2209.014150] LNetError: 73506:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2209.041852] LNet: Removed LNI 192.168.206.128@tcp [ 2210.128825] Key type .llcrypt unregistered [ 2210.131416] Key type ._llcrypt unregistered [ 2233.189675] Key type ._llcrypt registered [ 2233.194828] Key type .llcrypt registered [ 2233.274043] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 2234.306319] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2234.326380] alg: No test for adler32 (adler32-zlib) [ 2235.367217] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 2235.580603] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 2237.244129] Key type lgssc registered [ 2238.415656] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2246.201390] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2246.232802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2247.684410] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2252.093070] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 2256.873727] LustreError: 74914:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2256.896879] LustreError: 74914:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2259.930851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2260.346517] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2265.117086] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 2273.288855] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2279.333726] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2285.449531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2285.784992] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2292.566970] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 2292.989900] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2299.993633] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2308.597451] Lustre: Failing over lustre-MDT0000 [ 2308.982384] Lustre: server umount lustre-MDT0000 complete [ 2310.630953] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2310.638019] LustreError: 74910:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2310.655131] Lustre: Skipped 2 previous similar messages [ 2310.693735] LustreError: 74910:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2311.758700] LustreError: 74910:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2315.747366] LustreError: 74914:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2315.774718] LustreError: 74914:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2318.787347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2318.929860] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2319.186655] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2319.328857] LustreError: 78001:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2319.343561] LustreError: 78001:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.128@tcp: cfg command failed: rc = -9 [ 2319.351440] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2319.351440] [ 2319.377979] LustreError: MGC192.168.206.128@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2319.400275] LustreError: 77990:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2319.419476] LustreError: 77990:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -9 [ 2319.425935] LustreError: 77990:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 2319.431954] LustreError: 77990:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2319.491182] Lustre: server umount lustre-MDT0000 complete [ 2319.494638] LustreError: 77990:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2320.866091] LustreError: 74910:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2320.900238] LustreError: 74910:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2326.849428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2327.007208] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2327.020138] LustreError: 74907:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2327.328321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2327.386789] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2332.125978] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 2332.169563] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2332.648765] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2332.689024] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2332.753984] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2341.348414] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2341.359594] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2341.366804] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2342.881924] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2342.890441] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2342.917307] Lustre: Skipped 1 previous similar message [ 2345.551189] Lustre: server umount lustre-OST0000 complete [ 2353.121901] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2353.135168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2353.139076] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2353.140879] Lustre: Skipped 1 previous similar message [ 2358.251776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2358.261218] Lustre: Skipped 1 previous similar message [ 2363.360318] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2363.364827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2363.380256] Lustre: Skipped 1 previous similar message [ 2363.612577] Lustre: server umount lustre-MDT0000 complete [ 2366.626217] LustreError: 74896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788595950 with bad export cookie 10981328168085594931 [ 2366.637848] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2366.642532] LustreError: 74896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2366.866157] Lustre: server umount lustre-MDT0001 complete [ 2375.711190] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 2379.419827] Key type lgssc unregistered [ 2379.641904] LNet: 79781:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2379.645572] LNetError: 79781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2379.659316] LNet: Removed LNI 192.168.206.128@tcp [ 2380.249854] Key type .llcrypt unregistered [ 2380.252225] Key type ._llcrypt unregistered [ 2393.892970] Key type ._llcrypt registered [ 2393.903845] Key type .llcrypt registered [ 2394.056875] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 2394.367466] Key type .llcrypt unregistered [ 2394.370062] Key type ._llcrypt unregistered [ 2416.918501] Key type ._llcrypt registered [ 2416.921080] Key type .llcrypt registered [ 2417.021547] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 2417.903148] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2418.077830] alg: No test for adler32 (adler32-zlib) [ 2419.107098] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 2419.355263] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 2421.088174] Key type lgssc registered [ 2422.196313] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2429.206260] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2429.226731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2430.683934] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2435.281612] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 2440.164300] LustreError: 81674:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2440.205912] LustreError: 81674:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2443.852107] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2444.291319] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2448.531583] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 2455.532532] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2461.296330] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2468.015589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2468.378649] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2474.702839] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 2475.573081] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2482.397917] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2494.945753] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2494.953822] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2494.959720] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2495.971236] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2495.973631] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2495.975550] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2496.013956] Lustre: Skipped 1 previous similar message [ 2497.961672] Lustre: server umount lustre-OST0000 complete [ 2505.698161] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2505.705588] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2505.737032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2507.790887] Lustre: server umount lustre-MDT0000 complete [ 2511.338862] LustreError: 82401:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2511.377793] LustreError: 82401:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2512.075985] LustreError: 82408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788596096 with bad export cookie 9444927830262633718 [ 2512.077174] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2512.096602] LustreError: 82408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2512.593687] Lustre: server umount lustre-MDT0001 complete [ 2526.859727] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 2529.473321] Key type lgssc unregistered [ 2529.687684] LNet: 85206:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2529.692804] LNetError: 85206:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2529.704627] LNet: Removed LNI 192.168.206.128@tcp [ 2530.395952] Key type .llcrypt unregistered [ 2530.400088] Key type ._llcrypt unregistered [ 2551.234662] Key type ._llcrypt registered [ 2551.237102] Key type .llcrypt registered [ 2551.322602] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 2552.250575] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2552.325186] alg: No test for adler32 (adler32-zlib) [ 2553.388875] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 2553.603568] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 2555.288150] Key type lgssc registered [ 2556.642846] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2563.068309] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2563.081394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2564.435458] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2568.242392] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 2573.799465] LustreError: 86615:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2573.828404] LustreError: 86615:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2576.040622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2576.368996] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2580.894594] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 2587.745220] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2592.768345] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2598.431856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2598.823025] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2604.886781] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 2609.103818] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2612.393429] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2619.367632] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2619.372264] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2619.390047] Lustre: Skipped 1 previous similar message [ 2619.408528] Lustre: Skipped 1 previous similar message [ 2623.769631] Lustre: server umount lustre-OST0000 complete [ 2627.555853] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2627.565740] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2627.576505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2629.601454] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2629.602187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2629.630469] Lustre: Skipped 1 previous similar message [ 2634.729566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2634.738097] Lustre: Skipped 1 previous similar message [ 2638.819081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2638.825181] Lustre: Skipped 1 previous similar message [ 2641.376127] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2641.595502] Lustre: server umount lustre-MDT0000 complete [ 2643.937377] LustreError: 86616:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2643.970459] LustreError: 86616:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2645.328353] LustreError: 87349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788596229 with bad export cookie 11296972700166103348 [ 2645.343537] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2645.346084] LustreError: 87349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2645.679484] Lustre: server umount lustre-MDT0001 complete [ 2655.995484] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 2658.739291] Key type lgssc unregistered [ 2658.984616] LNet: 90148:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2658.990357] LNetError: 90148:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2659.010053] LNet: Removed LNI 192.168.206.128@tcp [ 2659.797413] Key type .llcrypt unregistered [ 2659.802791] Key type ._llcrypt unregistered [ 2681.911562] Key type ._llcrypt registered [ 2681.913646] Key type .llcrypt registered [ 2682.019710] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 2683.034682] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2683.090609] alg: No test for adler32 (adler32-zlib) [ 2684.187266] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 2684.503779] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 2686.256554] Key type lgssc registered [ 2687.481521] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2695.340901] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2695.372125] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2697.021895] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2701.763458] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 2707.431692] LustreError: 91560:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2707.442013] LustreError: 91560:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2708.846109] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2709.159047] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2712.911516] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 2718.994722] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2724.672894] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2731.076449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2731.479981] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2737.030314] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 2742.774025] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2743.214332] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2752.924896] Lustre: server umount lustre-OST0000 complete [ 2758.122075] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2758.126356] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2758.138739] Lustre: Skipped 1 previous similar message [ 2758.154660] Lustre: Skipped 1 previous similar message [ 2763.233259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2763.241832] Lustre: Skipped 1 previous similar message [ 2768.354464] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2768.362857] Lustre: Skipped 1 previous similar message [ 2769.376557] Lustre: 90870:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788596337/real 1788596337] req@00000000242b9532 x1875479120339840/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788596353 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2769.398134] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2770.400128] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2770.538600] Lustre: server umount lustre-MDT0000 complete [ 2771.424286] Lustre: 90868:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788596339/real 1788596339] req@00000000d6f6adb3 x1875479120340224/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788596355 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2773.411440] LustreError: 91539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788596357 with bad export cookie 14726363030589863844 [ 2773.418708] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2773.422739] LustreError: 91539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2773.636749] Lustre: server umount lustre-MDT0001 complete [ 2781.004556] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 2784.006700] Key type lgssc unregistered [ 2784.355526] LNet: 95093:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2784.360515] LNetError: 95093:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2784.377276] LNet: Removed LNI 192.168.206.128@tcp [ 2785.194273] Key type .llcrypt unregistered [ 2785.196517] Key type ._llcrypt unregistered [ 2810.162896] Key type ._llcrypt registered [ 2810.165224] Key type .llcrypt registered [ 2810.285321] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 2811.306898] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2811.388615] alg: No test for adler32 (adler32-zlib) [ 2812.500825] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 2812.775148] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 2814.448403] Key type lgssc registered [ 2815.348744] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2822.975668] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2823.000796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2838.560463] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2843.680377] LustreError: 96476:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.128@tcp: failed processing log, type 4: rc = -110 [ 2870.368195] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2876.130856] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 2882.923662] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2887.830863] Lustre: server umount lustre-OST0000 complete [ 2895.146172] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 04:21:18 (1788596478) [ 2900.586783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2900.896231] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2904.996691] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 2906.090279] LustreError: 98114:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2906.111057] LustreError: 98114:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2911.208884] LustreError: 98115:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2912.225099] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2912.521970] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2915.780464] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 2921.164438] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2926.132649] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2931.807775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2932.284367] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2938.067918] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 2945.558874] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2946.607147] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 2957.793438] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2957.797282] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2957.809574] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2961.894884] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2961.895932] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2961.897995] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2963.721928] Lustre: server umount lustre-OST0000 complete [ 2969.057226] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2969.065302] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2969.077687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2969.083050] Lustre: Skipped 1 previous similar message [ 2972.131390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2972.132414] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2973.561439] Lustre: server umount lustre-MDT0000 complete [ 2976.775055] LustreError: 98094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788596561 with bad export cookie 12214874328639419966 [ 2976.779524] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2976.788296] LustreError: 98094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2977.095857] Lustre: server umount lustre-MDT0001 complete [ 2986.830238] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 2989.374725] Key type lgssc unregistered [ 2989.748871] LNet: 101803:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2989.756906] LNetError: 101803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2989.767088] LNet: Removed LNI 192.168.206.128@tcp [ 2990.663454] Key type .llcrypt unregistered [ 2990.666192] Key type ._llcrypt unregistered [ 3025.894412] Key type ._llcrypt registered [ 3025.897456] Key type .llcrypt registered [ 3026.029760] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 3027.026579] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3027.067237] alg: No test for adler32 (adler32-zlib) [ 3028.110661] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 3028.320687] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 3029.986262] Key type lgssc registered [ 3031.196201] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3037.014663] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3037.029809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3038.413146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3042.510911] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 3047.912906] LustreError: 103963:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3047.933897] LustreError: 103963:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3049.061241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3049.323991] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3053.227272] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 3059.741946] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3064.508807] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3070.168494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3070.457711] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3075.566722] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 3082.060160] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3083.764912] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3095.011576] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3095.022415] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3095.037592] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3099.121939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3099.127260] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3099.134830] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3100.731489] Lustre: server umount lustre-OST0000 complete [ 3104.230643] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3104.237476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3104.240205] Lustre: Skipped 1 previous similar message [ 3104.246564] Lustre: Skipped 1 previous similar message [ 3109.368276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3109.381898] Lustre: Skipped 2 previous similar messages [ 3110.143612] Lustre: server umount lustre-MDT0000 complete [ 3113.750922] LustreError: 103942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788596697 with bad export cookie 13379146752744547943 [ 3113.766106] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3114.142784] Lustre: server umount lustre-MDT0001 complete [ 3124.551693] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 3127.118732] Key type lgssc unregistered [ 3127.495656] LNet: 107519:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3127.504053] LNetError: 107519:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3127.515603] LNet: Removed LNI 192.168.206.128@tcp [ 3128.566186] Key type .llcrypt unregistered [ 3128.567830] Key type ._llcrypt unregistered [ 3144.619866] Key type ._llcrypt registered [ 3144.621748] Key type .llcrypt registered [ 3144.721483] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 3145.669568] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3145.744198] alg: No test for adler32 (adler32-zlib) [ 3146.864719] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 3147.201683] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 3148.904204] Key type lgssc registered [ 3150.006532] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3157.218965] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3157.242911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3172.768379] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3177.888547] LustreError: 108735:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.128@tcp: failed processing log, type 4: rc = -110 [ 3204.512742] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3210.227535] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 3217.353128] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3223.068735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3223.335149] LustreError: MGC192.168.206.128@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3223.348716] LustreError: 109758:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3223.359873] LustreError: 109758:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [ 3223.365341] LustreError: 109758:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 3223.480968] Lustre: server umount lustre-MDT0000 complete [ 3223.483049] LustreError: 109758:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3244.000135] Lustre: 108072:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788596812/real 1788596812] req@000000009ff58e4b x1875479605808384/t0(0) o400->MGC192.168.206.128@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788596828 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3244.019258] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3244.031039] LustreError: 108747:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3264.587742] Lustre: server umount lustre-OST0000 complete [ 3273.367581] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_hostid [ 3282.127336] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 3315.554835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3315.747459] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3315.790920] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3315.887356] Lustre: lustre-MDT0000: new disk, initializing [ 3315.981743] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3316.004234] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3320.000246] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 3330.278554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3330.391354] Lustre: 112553:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3330.431526] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3330.435066] Lustre: Skipped 1 previous similar message [ 3330.488948] Lustre: lustre-MDT0001: new disk, initializing [ 3330.598394] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3330.666693] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3330.679538] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3334.876874] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 3343.152472] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3347.952944] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3354.139603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3354.418196] Lustre: lustre-OST0000: new disk, initializing [ 3354.424522] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3354.432113] Lustre: 114521:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3354.525059] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3359.916861] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 3361.317541] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3361.338224] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3361.407837] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3368.464603] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3372.767689] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3372.997557] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3377.349932] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3377.568798] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3381.735426] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3381.737473] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3381.749769] Lustre: Skipped 1 previous similar message [ 3381.756432] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3385.756397] Lustre: server umount lustre-OST0000 complete [ 3391.975155] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3391.976221] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3392.006595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3392.020264] Lustre: Skipped 2 previous similar messages [ 3397.089210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3397.097041] Lustre: Skipped 1 previous similar message [ 3402.216896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3402.221303] Lustre: Skipped 1 previous similar message [ 3403.232161] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3403.514737] Lustre: server umount lustre-MDT0000 complete [ 3406.848073] LustreError: 112546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788596991 with bad export cookie 3565083068619544489 [ 3406.850730] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3406.863898] LustreError: 112546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3407.334839] Lustre: server umount lustre-MDT0001 complete [ 3415.935206] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 04:29:59 (1788596999) [ 3418.576267] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3432.675946] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_hostid [ 3440.888906] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 3473.408036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3473.715892] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3473.756832] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3473.853541] Lustre: lustre-MDT0000: new disk, initializing [ 3473.959885] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3473.999496] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3478.173074] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 3488.389085] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3488.549793] Lustre: 119425:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3488.629710] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3488.633424] Lustre: Skipped 1 previous similar message [ 3488.710293] Lustre: lustre-MDT0001: new disk, initializing [ 3488.837233] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3488.883676] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3488.892723] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3493.486223] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 3503.308658] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3508.918350] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3515.513852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3515.785291] Lustre: lustre-OST0000: new disk, initializing [ 3515.790677] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3515.798103] Lustre: 121391:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3515.881477] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3521.105222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3521.115804] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3521.223597] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3522.004681] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 3530.952449] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3534.875653] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3535.068455] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3539.191698] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3539.467479] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3541.985602] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3541.985953] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3541.998665] LustreError: Skipped 1 previous similar message [ 3541.999498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3542.026493] Lustre: Skipped 2 previous similar messages [ 3547.543941] Lustre: server umount lustre-OST0000 complete [ 3555.307426] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3555.312971] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3555.327670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3555.354316] Lustre: Skipped 1 previous similar message [ 3555.376690] Lustre: Skipped 4 previous similar messages [ 3557.182098] Lustre: server umount lustre-MDT0000 complete [ 3560.764564] LustreError: 119415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788597145 with bad export cookie 3565083068619545371 [ 3560.784561] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3560.785764] LustreError: 119415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3561.192256] Lustre: server umount lustre-MDT0001 complete [ 3569.651544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3570.192856] LustreError: 123607:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3570.217483] LustreError: 123607:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3570.284942] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3575.273995] LustreError: 123608:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3575.607369] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 3580.394161] LustreError: 123607:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3584.859357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3589.127390] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 3596.104622] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3600.936224] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3607.211891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3613.570684] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 3620.735458] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3626.833544] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3631.587095] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3631.596637] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3631.613926] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3636.846805] Lustre: server umount lustre-OST0000 complete [ 3640.801305] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3640.814341] Lustre: Skipped 2 previous similar messages [ 3648.395846] Lustre: server umount lustre-MDT0000 complete [ 3651.043764] LustreError: 123603:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3651.058232] LustreError: 123603:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3651.506596] LustreError: 123590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788597235 with bad export cookie 3565083068619546253 [ 3651.523598] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3651.777664] Lustre: server umount lustre-MDT0001 complete [ 3660.034146] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 3662.345202] Key type lgssc unregistered [ 3662.632523] LNet: 127244:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3662.637812] LNetError: 127244:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3663.661723] LNet: Removed LNI 192.168.206.128@tcp [ 3664.527329] Key type .llcrypt unregistered [ 3664.530630] Key type ._llcrypt unregistered [ 3679.260441] Key type ._llcrypt registered [ 3679.262687] Key type .llcrypt registered [ 3679.356859] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_hostid [ 3691.683428] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 3692.538309] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3692.594713] alg: No test for adler32 (adler32-zlib) [ 3693.613044] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 3693.841533] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 3695.504154] Key type lgssc registered [ 3696.437503] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3724.654225] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3724.682034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3725.903621] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3725.939338] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3726.018372] Lustre: lustre-MDT0000: new disk, initializing [ 3726.095151] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3726.111126] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3730.342373] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 3739.768265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3739.902304] Lustre: 130376:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3740.016199] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3740.021284] Lustre: Skipped 1 previous similar message [ 3740.097619] Lustre: lustre-MDT0001: new disk, initializing [ 3740.235072] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3740.288098] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3740.302130] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3744.543935] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 3752.814761] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3757.745803] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3763.714233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3764.035883] Lustre: lustre-OST0000: new disk, initializing [ 3764.050411] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3764.059192] Lustre: 132344:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3764.138338] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3769.487560] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 3774.023126] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3774.046039] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3774.156886] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3778.231277] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3782.447107] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3782.633871] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3786.274085] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3786.442205] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3789.794778] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3789.799316] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3789.804643] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3789.829277] Lustre: Skipped 1 previous similar message [ 3794.535487] Lustre: server umount lustre-OST0000 complete [ 3800.035529] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3800.039802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3800.052121] Lustre: Skipped 1 previous similar message [ 3800.062840] Lustre: Skipped 1 previous similar message [ 3805.154480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3805.160546] Lustre: Skipped 2 previous similar messages [ 3810.277536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3810.284024] Lustre: Skipped 1 previous similar message [ 3812.320121] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3812.526121] Lustre: server umount lustre-MDT0000 complete [ 3815.998934] LustreError: 130366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788597400 with bad export cookie 18002915830026954356 [ 3816.007824] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3816.009717] LustreError: 130366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3816.290361] Lustre: server umount lustre-MDT0001 complete [ 3823.841417] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 04:36:47 (1788597407) [ 3829.362106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3829.691926] LustreError: 134755:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3829.709557] LustreError: 134755:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3829.797972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3833.705908] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 3834.853310] LustreError: 134756:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3838.951030] LustreError: 134755:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3841.838458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3842.236686] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3846.584348] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 3855.461818] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3860.201335] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3862.499070] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3862.503371] LustreError: Skipped 1 previous similar message [ 3862.505779] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3862.513413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3873.256338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3873.265847] Lustre: Skipped 5 previous similar messages [ 3876.320431] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3876.619638] Lustre: server umount lustre-MDT0000 complete [ 3878.371650] LustreError: 134755:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3878.398331] LustreError: 134755:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3879.738253] LustreError: 134735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788597463 with bad export cookie 18002915830026955245 [ 3879.740667] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3879.749165] LustreError: 134735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3880.114674] Lustre: server umount lustre-MDT0001 complete [ 3888.442716] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 04:37:52 (1788597472) [ 3894.458486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3910.117197] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3915.232657] LustreError: 137169:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.128@tcp: failed processing log, type 4: rc = -110 [ 3940.896881] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3947.644806] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 3955.049396] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3957.145329] Lustre: server umount lustre-OST0000 complete [ 3964.470581] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 04:39:08 (1788597548) [ 3970.155646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3970.541396] LustreError: 138614:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3970.605067] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3974.525924] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 3980.783233] LustreError: 138615:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3980.797838] LustreError: 138615:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3981.983519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3986.632518] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 3993.441249] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3998.476979] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4004.255764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4004.588982] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4004.592786] Lustre: Skipped 1 previous similar message [ 4009.733907] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 4015.788350] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4023.265186] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4023.282703] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4023.309750] Lustre: Skipped 1 previous similar message [ 4023.319961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4029.191438] Lustre: server umount lustre-MDT0000 complete [ 4031.473254] LustreError: 138614:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4031.504660] LustreError: 138614:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4032.600040] LustreError: 138594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788597616 with bad export cookie 18002915830026955910 [ 4032.610289] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4033.066627] Lustre: server umount lustre-MDT0001 complete [ 4047.094598] Lustre: server umount lustre-OST0000 complete [ 4054.127457] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 04:40:37 (1788597637) [ 4059.384258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4063.463431] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 4065.249350] LustreError: 142266:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4065.259151] LustreError: 142266:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4070.231147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4070.499750] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4070.505052] Lustre: Skipped 1 previous similar message [ 4073.518666] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 4078.189513] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4081.776475] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4086.359363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4090.756844] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 4096.308776] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4099.678404] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4103.984054] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4105.157260] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec [ 4108.805122] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4108.973897] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4111.331017] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4111.347234] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4111.362093] Lustre: Skipped 2 previous similar messages [ 4111.367187] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4111.371499] Lustre: Skipped 3 previous similar messages [ 4116.727272] Lustre: server umount lustre-OST0000 complete [ 4121.577371] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4121.589099] LustreError: Skipped 1 previous similar message [ 4121.592504] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4121.605428] Lustre: Skipped 1 previous similar message [ 4125.862615] Lustre: server umount lustre-MDT0000 complete [ 4129.124722] LustreError: 142247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788597713 with bad export cookie 18002915830026956981 [ 4129.134944] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4129.154503] LustreError: 142247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 4129.419525] Lustre: server umount lustre-MDT0001 complete [ 4137.013106] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 04:42:00 (1788597720) [ 4142.537242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4158.050450] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4163.168463] LustreError: 146456:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.128@tcp: failed processing log, type 4: rc = -110 [ 4194.667258] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 4201.124452] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4207.184996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4207.487216] LustreError: 146481:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4207.511647] LustreError: 146481:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4207.587700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4207.594839] Lustre: Skipped 2 previous similar messages [ 4207.659417] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4211.280559] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 4219.487528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4224.546364] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 4233.348512] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4238.899499] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4243.457737] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4243.630828] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4247.895467] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4248.087519] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4250.594535] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4250.603618] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4250.616088] Lustre: Skipped 1 previous similar message [ 4250.625232] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4250.630393] Lustre: Skipped 5 previous similar messages [ 4255.998708] Lustre: server umount lustre-MDT0000 complete [ 4259.331757] LustreError: 146462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788597843 with bad export cookie 18002915830026958003 [ 4259.337494] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4259.345241] LustreError: 146462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4259.658322] Lustre: server umount lustre-MDT0001 complete [ 4272.111940] Lustre: server umount lustre-OST0000 complete [ 4280.522552] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 04:44:24 (1788597864) [ 4287.084965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4302.689149] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4307.810586] LustreError: 150650:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.128@tcp: failed processing log, type 4: rc = -110 [ 4339.405489] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 4346.764099] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4353.772929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4354.263837] LustreError: 150675:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4354.289501] LustreError: 150675:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 4354.399165] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4357.889744] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 4365.522829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4371.078687] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 4377.378596] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4382.251096] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4387.919843] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4388.094081] Lustre: lustre-OST0001: new disk, initializing [ 4388.097049] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4388.102684] Lustre: 153522:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4393.600762] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 4397.649231] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4397.671140] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4397.793867] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4402.465745] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4406.105406] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4406.259872] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4409.511898] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4409.684400] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4412.385834] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4412.395630] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4412.406553] Lustre: Skipped 2 previous similar messages [ 4412.417466] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4412.429495] Lustre: Skipped 3 previous similar messages [ 4417.127905] Lustre: server umount lustre-OST0000 complete [ 4434.400275] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4434.618300] Lustre: server umount lustre-OST0001 complete [ 4452.320189] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4452.534627] Lustre: server umount lustre-MDT0000 complete [ 4456.035189] LustreError: 153521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788598040 with bad export cookie 18002915830026958955 [ 4456.039150] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4456.055501] LustreError: 153521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4456.323697] Lustre: server umount lustre-MDT0001 complete [ 4477.191396] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 04:47:40 (1788598060) [ 4478.593424] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4480.050832] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4481.776598] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 04:47:45 (1788598065) [ 4487.663938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4487.910229] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4487.937524] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4488.160185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4488.163838] Lustre: Skipped 5 previous similar messages [ 4493.245190] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 4500.203550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4500.285951] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4500.313765] Lustre: 157081:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4504.628489] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 4510.630281] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4514.895591] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4519.752219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4519.920602] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4525.108552] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 4528.223571] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4530.366286] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4534.285678] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4534.455325] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4538.270756] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4538.505566] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4543.458532] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4543.464227] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4543.483630] Lustre: Skipped 5 previous similar messages [ 4543.492495] LustreError: Skipped 3 previous similar messages [ 4546.317493] Lustre: server umount lustre-OST0000 complete [ 4568.249030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4575.487505] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 4581.619932] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4582.964309] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4582.993737] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4582.994422] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4585.558362] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4585.746925] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4590.166622] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4590.312666] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4595.271628] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9a6698116800.ost_server_uuid 50 [ 4596.643861] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9a6698116800.ost_server_uuid in FULL state after 0 sec [ 4596.849246] LustreError: 157091:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.206.28@tcp) returned error from blocking AST (req@000000008399c0d6 x1875480179624448 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 000000009aa6d44b/0xf9d734905a0381c5 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.206.28@tcp remote: 0xa4668c3cd7d06cc7 expref: 5 pid: 157091 timeout: 4696 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4596.876686] LustreError: lustre-MDT0000: A client on nid 192.168.206.28@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4596.886959] LustreError: 157079:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.206.28@tcp ns: mdt-lustre-MDT0000_UUID lock: 000000009aa6d44b/0xf9d734905a0381c5 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.206.28@tcp remote: 0xa4668c3cd7d06cc7 expref: 6 pid: 157091 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4607.233633] Lustre: server umount lustre-OST0000 complete [ 4613.089785] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4613.098749] LustreError: Skipped 1 previous similar message [ 4617.659098] Lustre: server umount lustre-MDT0000 complete [ 4618.721895] LustreError: 157095:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4618.746596] LustreError: 157095:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 4621.316161] LustreError: 157074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788598205 with bad export cookie 18002915830026960138 [ 4621.318612] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4621.335383] LustreError: 157074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4627.675520] Lustre: server umount lustre-MDT0001 complete [ 4637.763538] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 4640.928872] Key type lgssc unregistered [ 4641.248680] LNet: 163190:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4641.257315] LNetError: 163190:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4641.274469] LNet: Removed LNI 192.168.206.128@tcp [ 4642.258098] Key type .llcrypt unregistered [ 4642.261123] Key type ._llcrypt unregistered [ 4663.977659] Key type ._llcrypt registered [ 4663.980425] Key type .llcrypt registered [ 4664.092302] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 4665.269362] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4665.425655] alg: No test for adler32 (adler32-zlib) [ 4666.410115] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 4666.614686] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 4668.248123] Key type lgssc registered [ 4669.397543] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4677.823839] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4677.843631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4679.318277] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4683.950300] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 4688.869736] LustreError: 164591:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4688.877371] LustreError: 164591:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4691.911975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4692.263823] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4696.326854] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 4702.990330] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4708.012909] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4714.020377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4714.340974] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4720.131773] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 4724.608782] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 4726.855411] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4731.710859] Lustre: Failing over lustre-MDT0000 [ 4731.971561] Lustre: server umount lustre-MDT0000 complete [ 4733.410822] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4733.417868] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4733.437919] LustreError: 165319:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4733.482685] LustreError: 165319:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4734.948024] LustreError: 164591:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4734.956746] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4734.967779] LustreError: 164591:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4735.002671] Lustre: Skipped 1 previous similar message [ 4740.068338] LustreError: 164592:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4740.082249] LustreError: 164592:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4744.352755] LustreError: 164572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.206.28@tcp arrived at 1788598328 with bad export cookie 15607849468717307712 [ 4744.372789] LustreError: 164572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4745.194652] LustreError: 165319:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4745.213711] LustreError: 165319:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4750.697051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4750.863891] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4751.070148] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4751.112863] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4754.613787] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 4756.454452] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4756.461069] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4756.530774] Lustre: lustre-MDT0000: Denying connection for new client 0a8ef86c-2b2e-4dac-ba78-a60244dbc924 (at 192.168.206.28@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:09 [ 4761.621211] Lustre: lustre-MDT0000: Denying connection for new client 0a8ef86c-2b2e-4dac-ba78-a60244dbc924 (at 192.168.206.28@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:04 [ 4766.692746] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4766.697270] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4766.702059] Lustre: Skipped 1 previous similar message [ 4766.721149] Lustre: Skipped 1 previous similar message [ 4770.818791] Lustre: server umount lustre-OST0000 complete [ 4773.629044] LustreError: 168324:0:(ldlm_lib.c:3029:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 4773.635832] Lustre: 167674:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4773.642578] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 4773.668414] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 4773.683129] LustreError: 167674:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000075264d4f x1875481198635264/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4773.694629] LustreError: 167674:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 4773.700585] LustreError: 167674:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 4773.706568] Lustre: lustre-MDT0000: Recovery over after 0:17, of 2 clients 0 recovered and 2 were evicted. [ 4773.961263] Lustre: server umount lustre-MDT0000 complete [ 4776.930445] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4776.953897] LustreError: 164587:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4776.969545] LustreError: 164587:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4776.989316] LustreError: 164572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788598361 with bad export cookie 15607849468717307810 [ 4776.993474] LustreError: 163899:0:(import.c:751:ptlrpc_connect_import_locked()) can't connect to a closed import [ 4776.996352] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4776.997111] LustreError: 164572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4779.348204] Lustre: server umount lustre-MDT0001 complete [ 4789.077306] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 4791.783716] Key type lgssc unregistered [ 4792.111231] LNet: 169077:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4792.119917] LNetError: 169077:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4793.194071] LNet: Removed LNI 192.168.206.128@tcp [ 4794.191526] Key type .llcrypt unregistered [ 4794.194711] Key type ._llcrypt unregistered [ 4816.322343] Key type ._llcrypt registered [ 4816.324412] Key type .llcrypt registered [ 4816.431315] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 4817.482189] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4817.523181] alg: No test for adler32 (adler32-zlib) [ 4818.555520] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 4818.729658] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 4820.385947] Key type lgssc registered [ 4821.372493] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4828.558341] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4828.584990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4830.208732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4834.954928] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 4840.420777] LustreError: 170484:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4840.428493] LustreError: 170484:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4842.879909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4843.395703] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4848.946648] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 4855.517983] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4861.358637] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4867.178471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4867.589239] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4872.573558] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 4875.776810] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 4878.490963] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4885.985759] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4885.990656] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4886.007029] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4886.011398] Lustre: Skipped 1 previous similar message [ 4890.208345] Lustre: server umount lustre-OST0000 complete [ 4894.690071] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4894.697415] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4894.722263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4894.738277] Lustre: Skipped 1 previous similar message [ 4896.225893] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4896.227025] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4899.822235] Lustre: server umount lustre-MDT0000 complete [ 4901.347398] LustreError: 170484:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4901.360424] LustreError: 170484:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4903.152255] LustreError: 172328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788598487 with bad export cookie 10840577784667628706 [ 4903.155503] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4903.161524] LustreError: 172328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4903.424571] Lustre: server umount lustre-MDT0001 complete [ 4912.687786] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 4915.728169] Key type lgssc unregistered [ 4916.043860] LNet: 174013:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4916.059991] LNetError: 174013:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4916.081857] LNet: Removed LNI 192.168.206.128@tcp [ 4917.428868] Key type .llcrypt unregistered [ 4917.432270] Key type ._llcrypt unregistered [ 4952.712576] Key type ._llcrypt registered [ 4952.715308] Key type .llcrypt registered [ 4952.877986] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 4954.521812] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4954.597098] alg: No test for adler32 (adler32-zlib) [ 4955.910986] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 4956.203055] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 4957.880208] Key type lgssc registered [ 4959.332571] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4967.264687] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4967.314760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4968.750965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4973.650689] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 4978.146260] LustreError: 175715:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4978.165488] LustreError: 175715:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4982.400382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4982.828259] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4987.892475] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 4995.565513] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5001.278740] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5007.304173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5007.621354] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5013.463587] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 5013.745985] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5022.383518] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5037.541599] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5037.547248] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5037.552777] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5037.561119] Lustre: Skipped 1 previous similar message [ 5037.573811] Lustre: Skipped 1 previous similar message [ 5041.096355] Lustre: server umount lustre-OST0000 complete [ 5047.782931] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5047.825820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5047.836813] Lustre: Skipped 1 previous similar message [ 5051.235491] Lustre: server umount lustre-MDT0000 complete [ 5052.897170] LustreError: 176442:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5054.660123] LustreError: 175695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788598638 with bad export cookie 18151179324034326506 [ 5054.666568] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5054.668879] LustreError: 175695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5054.976613] Lustre: server umount lustre-MDT0001 complete [ 5065.490521] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 5068.568279] Key type lgssc unregistered [ 5068.848206] LNet: 179262:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5068.863351] LNetError: 179262:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5068.882547] LNet: Removed LNI 192.168.206.128@tcp [ 5070.130701] Key type .llcrypt unregistered [ 5070.136069] Key type ._llcrypt unregistered [ 5092.098881] Key type ._llcrypt registered [ 5092.105184] Key type .llcrypt registered [ 5092.217225] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 5093.425895] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5093.468562] alg: No test for adler32 (adler32-zlib) [ 5094.483961] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 5094.733897] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 5096.432193] Key type lgssc registered [ 5097.456050] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5106.392955] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5106.408064] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5107.816681] Lustre: *** cfs_fail_loc=135, val=0*** [ 5107.908496] LustreError: 180772:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5107.917784] LustreError: 180772:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.128@tcp: cfg command failed: rc = -2 [ 5107.930134] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5107.930134] [ 5107.945753] LustreError: MGC192.168.206.128@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5107.968671] LustreError: 180733:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5107.978119] LustreError: 180733:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [ 5107.985490] LustreError: 180733:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 5107.990562] LustreError: 180733:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5108.079574] Lustre: server umount lustre-MDT0000 complete [ 5108.081525] LustreError: 180733:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5114.147944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5114.449825] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5118.458802] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 5124.580332] LustreError: 181143:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5124.598988] LustreError: 181143:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5125.656622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5126.060113] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5130.245743] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 5136.567156] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5142.079687] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5144.548036] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5144.567228] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5144.572047] Lustre: Skipped 1 previous similar message [ 5149.667392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5149.681244] Lustre: Skipped 1 previous similar message [ 5151.715854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5151.720752] Lustre: Skipped 1 previous similar message [ 5156.834593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5156.840300] Lustre: Skipped 1 previous similar message [ 5158.368313] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5158.590888] Lustre: server umount lustre-MDT0000 complete [ 5161.957462] LustreError: 181144:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5161.980979] LustreError: 181144:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5161.997657] LustreError: 181877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788598746 with bad export cookie 7320298709620369299 [ 5161.999278] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5162.017832] LustreError: 181877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5162.461863] Lustre: server umount lustre-MDT0001 complete [ 5172.829439] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 5175.397226] Key type lgssc unregistered [ 5175.660661] LNet: 183464:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5175.670191] LNetError: 183464:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5175.683414] LNet: Removed LNI 192.168.206.128@tcp [ 5176.566291] Key type .llcrypt unregistered [ 5176.569716] Key type ._llcrypt unregistered [ 5196.993848] Key type ._llcrypt registered [ 5196.997661] Key type .llcrypt registered [ 5197.108598] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 5197.378512] Key type .llcrypt unregistered [ 5197.380197] Key type ._llcrypt unregistered [ 5213.122952] Key type ._llcrypt registered [ 5213.126226] Key type .llcrypt registered [ 5213.223795] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 5214.131052] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5214.170629] alg: No test for adler32 (adler32-zlib) [ 5215.215620] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 5215.512336] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 5217.216185] Key type lgssc registered [ 5218.329397] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5225.615951] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5225.628348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5241.120528] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5246.240614] LustreError: 185587:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.128@tcp: failed processing log, type 4: rc = -110 [ 5272.992185] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5278.505246] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 5285.776767] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5291.428914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5291.607059] LustreError: 184923:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000239f0d4a x1875481774262528/t0(0) o250->MGC192.168.206.128@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5292.079397] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5292.204058] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5297.124899] LustreError: 185612:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5297.165584] LustreError: 185612:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5298.923318] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 5302.273066] LustreError: 185613:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5307.361917] LustreError: 185612:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5307.391653] LustreError: 185612:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5308.338518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5308.730303] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5313.527551] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 5318.767269] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5323.711652] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5326.255542] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5338.082216] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5338.090739] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5338.104031] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5339.632280] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5339.634630] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5339.634673] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5342.654442] Lustre: server umount lustre-OST0000 complete [ 5349.859247] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5349.859653] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5349.861449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5349.861455] Lustre: Skipped 1 previous similar message [ 5349.899983] Lustre: Skipped 1 previous similar message [ 5352.953129] Lustre: server umount lustre-MDT0000 complete [ 5354.986911] LustreError: 186642:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5356.507735] LustreError: 185594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788598940 with bad export cookie 11967979348493747018 [ 5356.523730] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5356.532298] LustreError: 185594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5356.992641] Lustre: server umount lustre-MDT0001 complete [ 5365.898760] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 5368.195234] Key type lgssc unregistered [ 5368.489368] LNet: 189563:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5368.495291] LNetError: 189563:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5368.518636] LNet: Removed LNI 192.168.206.128@tcp [ 5369.217187] Key type .llcrypt unregistered [ 5369.220778] Key type ._llcrypt unregistered [ 5390.375792] Key type ._llcrypt registered [ 5390.377527] Key type .llcrypt registered [ 5390.496368] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 5391.620097] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5391.669391] alg: No test for adler32 (adler32-zlib) [ 5392.738817] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 5392.976684] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 5394.656435] Key type lgssc registered [ 5395.800750] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5401.478764] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5401.494619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5402.813316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5406.371449] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 5412.406667] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5412.769577] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5416.888876] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 5424.713435] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5431.494221] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5440.551039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5441.055704] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5447.426726] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 5449.279197] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 5453.575803] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5457.730508] Lustre: Failing over lustre-MDT0000 [ 5457.951620] Lustre: server umount lustre-MDT0000 complete [ 5458.918727] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5458.931561] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5458.958765] LustreError: 190951:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5458.974598] LustreError: 190951:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5460.527540] LustreError: 190947:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5462.499284] LustreError: 191679:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5462.501506] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5462.541796] LustreError: 191679:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5462.576864] Lustre: Skipped 1 previous similar message [ 5465.609634] LustreError: 190948:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5465.628392] LustreError: 190948:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5470.731896] LustreError: 190946:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5470.747406] LustreError: 190946:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5475.105457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5475.216952] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5475.400020] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5475.444319] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5475.858364] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5479.612204] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 5480.428629] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5480.448632] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 5480.501887] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 5483.901962] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 5487.380193] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 5495.785188] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5495.798711] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5495.812227] Lustre: Skipped 1 previous similar message [ 5497.300821] Lustre: server umount lustre-OST0000 complete [ 5500.898582] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5500.900380] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5500.917408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5500.923512] Lustre: Skipped 2 previous similar messages [ 5506.018882] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5506.030956] Lustre: Skipped 1 previous similar message [ 5511.145348] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5511.154653] Lustre: Skipped 1 previous similar message [ 5515.232118] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5515.401965] Lustre: server umount lustre-MDT0000 complete [ 5516.260426] LustreError: 191679:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5516.271726] LustreError: 191679:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 5518.420566] LustreError: 194527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788599102 with bad export cookie 15633330114045793139 [ 5518.427222] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5518.429531] LustreError: 194527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5518.674076] Lustre: server umount lustre-MDT0001 complete [ 5527.850531] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 5530.381610] Key type lgssc unregistered [ 5530.676968] LNet: 195737:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5530.683601] LNetError: 195737:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5530.715670] LNet: Removed LNI 192.168.206.128@tcp [ 5531.517281] Key type .llcrypt unregistered [ 5531.522344] Key type ._llcrypt unregistered [ 5554.953824] Key type ._llcrypt registered [ 5554.957440] Key type .llcrypt registered [ 5555.077741] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 5556.151583] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5556.219034] alg: No test for adler32 (adler32-zlib) [ 5557.291017] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 5557.568930] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 5559.312190] Key type lgssc registered [ 5560.556382] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5567.534822] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5567.553316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5569.019269] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5572.821521] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 5579.243596] LustreError: 197149:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5579.261306] LustreError: 197149:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5581.082885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5581.449640] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5586.306553] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 5593.360289] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5598.958536] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5604.717751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5605.024529] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5610.403782] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 5614.275486] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 5617.013379] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5619.586924] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 5628.122701] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 5640.677254] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5647.330339] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5647.337843] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5647.351435] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5650.405972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5650.406597] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5650.407070] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5650.412966] Lustre: Skipped 1 previous similar message [ 5653.547746] Lustre: server umount lustre-OST0000 complete [ 5658.081348] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5658.086775] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5658.099518] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5660.642920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5660.644893] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5663.741970] Lustre: server umount lustre-MDT0000 complete [ 5665.764742] LustreError: 197149:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5665.782083] LustreError: 197149:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5667.192374] LustreError: 197129:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788599251 with bad export cookie 4895919581642974672 [ 5667.195922] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5667.205130] LustreError: 197129:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5667.450140] Lustre: server umount lustre-MDT0001 complete [ 5676.975731] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 5679.622431] Key type lgssc unregistered [ 5679.840679] LNet: 200999:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5679.848245] LNetError: 200999:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5679.863114] LNet: Removed LNI 192.168.206.128@tcp [ 5680.580941] Key type .llcrypt unregistered [ 5680.584047] Key type ._llcrypt unregistered [ 5696.971681] Key type ._llcrypt registered [ 5696.974593] Key type .llcrypt registered [ 5697.076175] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 5697.782200] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5697.842736] alg: No test for adler32 (adler32-zlib) [ 5698.823578] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 5699.016426] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 5700.672147] Key type lgssc registered [ 5701.642808] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5708.999744] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5709.016806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5710.552715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5716.264732] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 5720.034971] LustreError: 202269:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5720.044414] LustreError: 202269:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5724.589762] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5725.127457] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5729.539615] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 5736.035281] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5741.488845] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5746.992362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5747.405535] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5752.801875] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 5755.564736] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 5759.220430] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5763.760442] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 5773.867477] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5783.120474] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 5787.798136] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 5796.833618] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5796.841207] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5796.841849] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5796.846640] Lustre: Skipped 1 previous similar message [ 5796.863668] Lustre: Skipped 1 previous similar message [ 5801.898383] Lustre: server umount lustre-OST0000 complete [ 5807.073732] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5807.076917] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5807.097353] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5807.109498] Lustre: Skipped 1 previous similar message [ 5812.195558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5812.206103] Lustre: Skipped 1 previous similar message [ 5817.318218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5817.323248] Lustre: Skipped 1 previous similar message [ 5819.360118] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5819.474164] Lustre: server umount lustre-MDT0000 complete [ 5822.433340] LustreError: 202270:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5822.442627] LustreError: 202270:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5822.631309] LustreError: 204115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788599406 with bad export cookie 6843743852514687875 [ 5822.635053] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5822.642368] LustreError: 204115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5822.980742] Lustre: server umount lustre-MDT0001 complete [ 5833.398873] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 5836.057472] Key type lgssc unregistered [ 5836.294715] LNet: 207193:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5836.300458] LNetError: 207193:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5836.315588] LNet: Removed LNI 192.168.206.128@tcp [ 5837.232232] Key type .llcrypt unregistered [ 5837.235242] Key type ._llcrypt unregistered [ 5861.511624] Key type ._llcrypt registered [ 5861.517466] Key type .llcrypt registered [ 5861.666659] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 5862.679358] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5862.744125] alg: No test for adler32 (adler32-zlib) [ 5863.817184] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 5864.055980] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 5865.704108] Key type lgssc registered [ 5866.806335] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5875.144913] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5875.170404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5876.713940] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5880.712846] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 5885.930931] LustreError: 208603:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5885.946986] LustreError: 208603:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5888.597476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5889.030654] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5893.083489] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 5899.961846] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5906.804166] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5913.948926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5914.458904] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5920.442961] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 5921.650994] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 5926.219877] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5941.221488] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5941.225201] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5941.230363] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5942.243561] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5942.244561] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5942.246784] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5942.290430] Lustre: Skipped 1 previous similar message [ 5945.592538] Lustre: server umount lustre-OST0000 complete [ 5950.435536] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5950.444648] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5950.475140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5952.482310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5952.483332] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5956.546622] Lustre: server umount lustre-MDT0000 complete [ 5957.603493] LustreError: 210803:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5957.629238] LustreError: 210803:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5961.238342] LustreError: 208583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788599545 with bad export cookie 5167539646014938919 [ 5961.242689] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5961.269249] LustreError: 208583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5961.578486] Lustre: server umount lustre-MDT0001 complete [ 5972.436459] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 5975.320439] Key type lgssc unregistered [ 5975.608599] LNet: 212285:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5975.621564] LNetError: 212285:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5975.642675] LNet: Removed LNI 192.168.206.128@tcp [ 5976.620877] Key type .llcrypt unregistered [ 5976.624048] Key type ._llcrypt unregistered [ 5996.136576] Key type ._llcrypt registered [ 5996.141387] Key type .llcrypt registered [ 5996.284720] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 5997.477940] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5997.492332] alg: No test for adler32 (adler32-zlib) [ 5998.612464] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 5998.937117] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 6000.624297] Key type lgssc registered [ 6001.659189] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6009.325060] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6009.341287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6010.735041] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6014.846841] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 6020.066450] LustreError: 213549:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6020.088636] LustreError: 213549:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6024.160716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6024.555696] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6029.247683] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 6036.421834] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6041.618521] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6048.146611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6048.446093] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6054.064320] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 6055.599450] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6060.441280] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6075.361954] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6075.377271] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6075.402841] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6076.389715] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6076.390346] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6078.616967] Lustre: server umount lustre-OST0000 complete [ 6086.115194] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6086.130428] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6086.154836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6086.163107] Lustre: Skipped 1 previous similar message [ 6091.760567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6091.768373] Lustre: Skipped 2 previous similar messages [ 6096.352508] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6096.506171] Lustre: server umount lustre-MDT0000 complete [ 6096.865667] LustreError: 214983:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6096.894390] LustreError: 214983:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6099.736166] LustreError: 215395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788599683 with bad export cookie 14449212272926351534 [ 6099.741920] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6099.748991] LustreError: 215395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6099.989256] Lustre: server umount lustre-MDT0001 complete [ 6108.820141] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 6111.294773] Key type lgssc unregistered [ 6111.520359] LNet: 217231:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6111.526603] LNetError: 217231:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6111.542726] LNet: Removed LNI 192.168.206.128@tcp [ 6112.094786] Key type .llcrypt unregistered [ 6112.097208] Key type ._llcrypt unregistered [ 6129.726761] Key type ._llcrypt registered [ 6129.729975] Key type .llcrypt registered [ 6129.815157] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 6130.509742] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6130.532545] alg: No test for adler32 (adler32-zlib) [ 6131.519922] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 6131.653620] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 6133.280261] Key type lgssc registered [ 6134.518993] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6142.214743] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6142.239540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6143.597713] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6147.038905] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 6153.196281] LustreError: 218490:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6153.215106] LustreError: 218490:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6154.210610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6154.727338] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6158.386577] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 6164.562159] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6169.699555] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6175.396392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6175.862768] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6181.735407] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 6188.146703] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6188.599304] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6196.534069] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6196.602139] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6196.743051] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6202.726827] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 6209.220941] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6210.488286] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6222.042375] Lustre: Permanently deactivating lustre-OST0001 [ 6222.053403] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6229.051332] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6236.065233] Lustre: Permanently reactivating lustre-OST0001 [ 6236.105882] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6236.121831] Lustre: Skipped 2 previous similar messages [ 6236.672298] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6236.688575] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6236.698404] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6236.712409] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6236.740222] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6249.449513] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6249.462268] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6249.487464] Lustre: Skipped 2 previous similar messages [ 6249.513054] Lustre: Skipped 1 previous similar message [ 6254.563285] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6254.899686] Lustre: server umount lustre-OST0001 complete [ 6259.684782] LustreError: 220311:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6259.690090] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6259.717551] LustreError: 220311:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6259.759268] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6259.766904] Lustre: Skipped 2 previous similar messages [ 6264.582553] Lustre: server umount lustre-OST0000 complete [ 6269.941981] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6269.944499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6269.948106] Lustre: Skipped 2 previous similar messages [ 6269.955467] Lustre: Skipped 1 previous similar message [ 6273.617708] Lustre: server umount lustre-MDT0000 complete [ 6275.042676] LustreError: 218490:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6275.068222] LustreError: 218490:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6276.798529] LustreError: 218472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788599861 with bad export cookie 9607807647052124644 [ 6276.799703] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6276.809259] LustreError: 218472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6277.126679] Lustre: server umount lustre-MDT0001 complete [ 6289.255268] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 6292.331742] Key type lgssc unregistered [ 6292.563594] LNet: 223551:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6292.571881] LNetError: 223551:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6292.595617] LNet: Removed LNI 192.168.206.128@tcp [ 6293.326521] Key type .llcrypt unregistered [ 6293.330922] Key type ._llcrypt unregistered [ 6331.332352] Key type ._llcrypt registered [ 6331.334691] Key type .llcrypt registered [ 6331.424605] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 6332.575886] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6332.619429] alg: No test for adler32 (adler32-zlib) [ 6333.748335] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 6333.960224] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 6335.608321] Key type lgssc registered [ 6336.515966] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6343.551656] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6343.567569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6344.791561] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6344.811584] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6345.058822] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6349.111147] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 6357.298960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6357.426247] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6357.458830] Lustre: 225754:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6357.787439] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6361.858043] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 6368.816765] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6374.478608] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6380.959033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6381.124978] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6381.335551] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6387.818726] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 6389.899730] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 6395.718788] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6398.457840] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6406.120868] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6416.198449] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6424.969276] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6440.267094] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6440.278747] Lustre: Skipped 1 previous similar message [ 6460.422434] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6460.431219] Lustre: Skipped 1 previous similar message [ 6493.850991] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6493.857675] Lustre: Skipped 4 previous similar messages [ 6567.689381] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 6567.695720] Lustre: Skipped 8 previous similar messages [ 6574.050175] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6574.057198] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6574.064825] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6574.563244] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6574.573974] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6574.595934] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6577.640161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6577.647600] Lustre: Skipped 1 previous similar message [ 6580.248153] Lustre: server umount lustre-OST0000 complete [ 6587.877446] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6587.891379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6587.892756] Lustre: Skipped 1 previous similar message [ 6587.902146] Lustre: Skipped 1 previous similar message [ 6592.994083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6592.997127] Lustre: Skipped 1 previous similar message [ 6598.112208] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6598.433854] Lustre: server umount lustre-MDT0000 complete [ 6602.139693] LustreError: 228494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788600186 with bad export cookie 14321439265690496677 [ 6602.149860] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6602.155446] LustreError: 228494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6602.530722] Lustre: server umount lustre-MDT0001 complete [ 6612.880196] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 6615.978944] Key type lgssc unregistered [ 6616.311705] LNet: 230557:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6616.330056] LNetError: 230557:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6616.351388] LNet: Removed LNI 192.168.206.128@tcp [ 6617.152884] Key type .llcrypt unregistered [ 6617.155467] Key type ._llcrypt unregistered [ 6637.162358] Key type ._llcrypt registered [ 6637.164874] Key type .llcrypt registered [ 6637.271748] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 6638.379530] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6638.626165] alg: No test for adler32 (adler32-zlib) [ 6639.844630] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 6640.291520] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 6642.080694] Key type lgssc registered [ 6643.568440] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6650.741743] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6650.759391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6652.147816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6656.759925] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 6662.631494] LustreError: 231948:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6662.649551] LustreError: 231948:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6665.170591] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6665.716459] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6670.250224] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 6678.946928] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6685.585496] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6693.324790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6693.612799] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6697.740901] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 6699.260779] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_default_debug -1 all [ 6706.518335] Lustre: DEBUG MARKER: oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6723.558655] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6723.575910] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6723.598041] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6725.606108] LustreError: 233772:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6725.620957] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6725.626868] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6725.748765] Lustre: server umount lustre-OST0000 complete [ 6730.724314] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6730.730706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6730.743332] Lustre: Skipped 1 previous similar message [ 6730.756843] Lustre: Skipped 1 previous similar message [ 6735.069467] Lustre: server umount lustre-MDT0000 complete [ 6735.842730] LustreError: 231944:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6735.855154] LustreError: 231944:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6737.921928] LustreError: 231929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788600322 with bad export cookie 1080980602873483951 [ 6737.924355] LustreError: MGC192.168.206.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6737.931576] LustreError: 231929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6738.206631] Lustre: server umount lustre-MDT0001 complete [ 6746.613752] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 6749.227710] Key type lgssc unregistered [ 6749.528077] LNet: 235637:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6749.531972] LNetError: 235637:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6749.548829] LNet: Removed LNI 192.168.206.128@tcp [ 6750.345556] Key type .llcrypt unregistered [ 6750.347930] Key type ._llcrypt unregistered [ 6771.887774] Key type ._llcrypt registered [ 6771.890437] Key type .llcrypt registered [ 6772.008439] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 6773.130700] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6773.290781] alg: No test for adler32 (adler32-zlib) [ 6774.578949] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 6774.936355] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 6776.736339] Key type lgssc registered [ 6777.939083] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7048.225486] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 7050.975602] Key type lgssc unregistered [ 7051.260828] LNet: 237654:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7051.270554] LNetError: 237654:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7051.302306] LNet: Removed LNI 192.168.206.128@tcp [ 7052.050860] Key type .llcrypt unregistered [ 7052.053970] Key type ._llcrypt unregistered [ 7075.181716] Key type ._llcrypt registered [ 7075.184505] Key type .llcrypt registered [ 7075.335163] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing set_hostid [ 7090.002821] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing load_modules_local [ 7091.026876] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7091.057975] alg: No test for adler32 (adler32-zlib) [ 7092.137431] Lustre: Lustre: Build Version: 2.17.58_2_gae7f7f4 [ 7092.361226] LNet: Added LNI 192.168.206.128@tcp [8/256/0/180] [ 7094.032220] Key type lgssc registered [ 7095.381859] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7119.606485] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 6823 sec ========== 05:31:43 (1788600703) [ 7120.971155] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 05:31:44 (1788600704) === [ 7123.946893] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 05:31:47 (1788600707) === [ 7159.455287] Lustre: DEBUG MARKER: oleg628-server.virtnet: executing unload_modules_local [ 7162.351046] Key type lgssc unregistered [ 7162.595775] LNet: 242886:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7162.611969] LNetError: 242886:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7162.627703] LNet: Removed LNI 192.168.206.128@tcp [ 7163.344262] Key type .llcrypt unregistered [ 7163.348053] Key type ._llcrypt unregistered