[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 588698170 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001021] APIC: Switch to symmetric I/O mode setup [ 0.003050] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.008018] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010141] LSM: Security Framework initializing [ 0.012042] Yama: becoming mindful. [ 0.012916] SELinux: Initializing. [ 0.013073] *** VALIDATE selinux *** [ 0.020724] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025568] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026143] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027113] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028099] *** VALIDATE tmpfs *** [ 0.030450] *** VALIDATE proc *** [ 0.031222] *** VALIDATE cgroup *** [ 0.032009] *** VALIDATE cgroup2 *** [ 0.033244] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035117] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037025] Spectre V2 : User space: Vulnerable [ 0.039004] Speculative Store Bypass: Vulnerable [ 0.042176] debug: unmapping init [mem 0xffffffffb6e59000-0xffffffffb6e60fff] [ 0.044761] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045620] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046022] ... version: 2 [ 0.047011] ... bit width: 48 [ 0.048009] ... generic registers: 4 [ 0.049011] ... value mask: 0000ffffffffffff [ 0.050010] ... max period: 00007fffffffffff [ 0.051009] ... fixed-purpose events: 3 [ 0.052009] ... event mask: 000000070000000f [ 0.053289] rcu: Hierarchical SRCU implementation. [ 0.055304] smp: Bringing up secondary CPUs ... [ 0.056535] x86: Booting SMP configuration: [ 0.057023] .... node #0, CPUs: #1 #2 #3 [ 0.061011] smp: Brought up 1 node, 4 CPUs [ 0.063017] smpboot: Max logical packages: 1 [ 0.064011] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.136281] node 0 deferred pages initialised in 69ms [ 0.140097] devtmpfs: initialized [ 0.141245] x86/mm: Memory block size: 128MB [ 0.144790] gcov: version magic: 0x41383552 [ 0.148289] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.151086] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.154296] pinctrl core: initialized pinctrl subsystem [ 0.156174] [ 0.156745] ************************************************************* [ 0.158010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.160011] ** ** [ 0.163011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.165013] ** ** [ 0.167011] ** This means that this kernel is built to expose internal ** [ 0.169009] ** IOMMU data structures, which may compromise security on ** [ 0.171009] ** your system. ** [ 0.173009] ** ** [ 0.175008] ** If you see this message and you are not debugging the ** [ 0.176009] ** kernel, report this immediately to your vendor! ** [ 0.178053] ** ** [ 0.180009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.182035] ************************************************************* [ 0.185668] NET: Registered protocol family 16 [ 0.187437] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.191061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.194061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.198553] cpuidle: using governor menu [ 0.200104] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.202389] PCI: Using configuration type 1 for base access [ 0.205136] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.215297] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.217028] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.221058] cryptd: max_cpu_qlen set to 1000 [ 0.224052] ACPI: Added _OSI(Module Device) [ 0.225009] ACPI: Added _OSI(Processor Device) [ 0.227014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.229013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.234041] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.241556] ACPI: Interpreter enabled [ 0.243054] ACPI: PM: (supports S0 S3 S4 S5) [ 0.245013] ACPI: Using IOAPIC for interrupt routing [ 0.247088] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.251405] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.261694] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.264031] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.266019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.269112] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.274916] acpiphp: Slot [2] registered [ 0.276090] acpiphp: Slot [5] registered [ 0.277091] acpiphp: Slot [6] registered [ 0.278077] acpiphp: Slot [7] registered [ 0.279120] acpiphp: Slot [8] registered [ 0.281117] acpiphp: Slot [9] registered [ 0.282081] acpiphp: Slot [10] registered [ 0.283084] acpiphp: Slot [3] registered [ 0.285262] acpiphp: Slot [4] registered [ 0.286071] acpiphp: Slot [11] registered [ 0.287068] acpiphp: Slot [12] registered [ 0.289114] acpiphp: Slot [13] registered [ 0.290102] acpiphp: Slot [14] registered [ 0.291072] acpiphp: Slot [15] registered [ 0.293124] acpiphp: Slot [16] registered [ 0.294086] acpiphp: Slot [17] registered [ 0.296095] acpiphp: Slot [18] registered [ 0.298091] acpiphp: Slot [19] registered [ 0.299088] acpiphp: Slot [20] registered [ 0.300154] acpiphp: Slot [21] registered [ 0.302091] acpiphp: Slot [22] registered [ 0.303146] acpiphp: Slot [23] registered [ 0.305076] acpiphp: Slot [24] registered [ 0.307101] acpiphp: Slot [25] registered [ 0.308118] acpiphp: Slot [26] registered [ 0.310100] acpiphp: Slot [27] registered [ 0.312095] acpiphp: Slot [28] registered [ 0.314087] acpiphp: Slot [29] registered [ 0.315074] acpiphp: Slot [30] registered [ 0.317084] acpiphp: Slot [31] registered [ 0.318058] PCI host bridge to bus 0000:00 [ 0.320019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.322021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.324021] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.327020] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.329028] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.332043] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.334172] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.338982] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.342219] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.355009] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.362012] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.364018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.366015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.370016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.373464] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.375668] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.378041] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.380758] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.385014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.399829] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.405837] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.413250] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.450013] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.464019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.492019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.531691] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.566014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.580016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.664015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.678170] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.690020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.716018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.793022] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.806217] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.833015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.840013] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.863013] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.878110] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.887017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.897020] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.917024] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.940514] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.956018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.967022] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 1.010017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 1.026460] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 1.029356] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 1.031417] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 1.034388] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 1.037220] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 1.042224] iommu: Default domain type: Passthrough [ 1.044570] SCSI subsystem initialized [ 1.047118] ACPI: bus type USB registered [ 1.048111] usbcore: registered new interface driver usbfs [ 1.050064] usbcore: registered new interface driver hub [ 1.052111] usbcore: registered new device driver usb [ 1.054176] pps_core: LinuxPPS API ver. 1 registered [ 1.056011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.059059] PTP clock support registered [ 1.062044] EDAC MC: Ver: 3.0.0 [ 1.063538] PCI: Using ACPI for IRQ routing [ 1.065838] NetLabel: Initializing [ 1.067011] NetLabel: domain hash size = 128 [ 1.068010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.070091] NetLabel: unlabeled traffic allowed by default [ 1.072129] vgaarb: loaded [ 1.073291] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.075016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.083374] clocksource: Switched to clocksource kvm-clock [ 1.187833] VFS: Disk quotas dquot_6.6.0 [ 1.189731] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.192606] *** VALIDATE ramfs *** [ 1.193918] *** VALIDATE hugetlbfs *** [ 1.196242] pnp: PnP ACPI init [ 1.198742] pnp: PnP ACPI: found 6 devices [ 1.258214] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.261068] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.262799] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.264454] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.266413] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.268439] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.270964] NET: Registered protocol family 2 [ 1.273321] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.277402] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.280587] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.285315] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.288150] TCP: Hash tables configured (established 65536 bind 65536) [ 1.290560] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.293457] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.296045] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.298538] NET: Registered protocol family 1 [ 1.302079] RPC: Registered named UNIX socket transport module. [ 1.303758] RPC: Registered udp transport module. [ 1.305255] RPC: Registered tcp transport module. [ 1.306553] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.308832] NET: Registered protocol family 44 [ 1.310418] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.312047] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.313627] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.315391] PCI: CLS 0 bytes, default 64 [ 1.317334] Unpacking initramfs... [ 2.684471] debug: unmapping init [mem 0xffff8f883cc54000-0xffff8f883ffbffff] [ 2.688742] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.691192] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.694430] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 3.217122] Initialise system trusted keyrings [ 3.219225] Key type blacklist registered [ 3.220992] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.229148] zbud: loaded [ 3.231918] *** VALIDATE nfs *** [ 3.232887] *** VALIDATE nfs4 *** [ 3.234170] pstore: using deflate compression [ 3.237216] Platform Keyring initialized [ 3.355753] NET: Registered protocol family 38 [ 3.358347] Key type asymmetric registered [ 3.360091] Asymmetric key parser 'x509' registered [ 3.361686] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.364707] io scheduler mq-deadline registered [ 3.366195] io scheduler kyber registered [ 3.367496] io scheduler bfq registered [ 3.369064] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.371610] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.374357] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.376726] ACPI: Power Button [PWRF] [ 3.382482] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.388301] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.442440] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.472559] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.494753] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.524743] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.557777] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.565803] Non-volatile memory driver v1.3 [ 3.567278] Linux agpgart interface v0.103 [ 3.594256] virtio_blk virtio1: [vda] 146016 512-byte logical blocks (74.8 MB/71.3 MiB) [ 3.596991] vda: detected capacity change from 0 to 74760192 [ 3.610438] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.613661] vdb: detected capacity change from 0 to 1073741824 [ 3.629680] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.633126] vdc: detected capacity change from 0 to 2621440000 [ 3.648759] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.651815] vdd: detected capacity change from 0 to 2621440000 [ 3.675469] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.678150] vde: detected capacity change from 0 to 4294967296 [ 3.700382] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.703287] vdf: detected capacity change from 0 to 4294967296 [ 3.711194] libphy: Fixed MDIO Bus: probed [ 3.715484] usbcore: registered new interface driver usbserial_generic [ 3.717710] usbserial: USB Serial support registered for generic [ 3.720155] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.724326] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.726147] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.728176] mousedev: PS/2 mouse device common for all mice [ 3.731221] rtc_cmos 00:05: RTC can wake from S4 [ 3.732408] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.737026] rtc_cmos 00:05: registered as rtc0 [ 3.741506] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.742699] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.747312] intel_pstate: CPU model not supported [ 3.750731] hid: raw HID events driver (C) Jiri Kosina [ 3.755040] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.757738] usbcore: registered new interface driver usbhid [ 3.757744] usbhid: USB HID core driver [ 3.757858] drop_monitor: Initializing network drop monitor service [ 3.758015] Initializing XFRM netlink socket [ 3.758376] NET: Registered protocol family 10 [ 3.760289] Segment Routing with IPv6 [ 3.773763] NET: Registered protocol family 17 [ 3.776155] mpls_gso: MPLS GSO support [ 3.783140] RAS: Correctable Errors collector initialized. [ 3.785146] AVX version of gcm_enc/dec engaged. [ 3.786649] AES CTR mode by8 optimization enabled [ 3.877673] sched_clock: Marking stable (3877654848, 0)->(4729436952, -851782104) [ 3.881216] registered taskstats version 1 [ 3.883187] Loading compiled-in X.509 certificates [ 3.885178] zswap: loaded using pool lzo/zbud [ 3.910078] Key type big_key registered [ 3.923703] Key type encrypted registered [ 3.925969] ima: No TPM chip found, activating TPM-bypass! [ 3.927918] ima: Allocated hash algorithm: sha1 [ 3.929292] ima: No architecture policies found [ 3.931270] evm: Initialising EVM extended attributes: [ 3.933210] evm: security.selinux [ 3.934567] evm: security.ima [ 3.935906] evm: security.capability [ 3.937157] evm: HMAC attrs: 0x1 [ 3.939852] rtc_cmos 00:05: setting system clock to 2026-08-06 16:48:53 UTC (1786034933) [ 3.945190] debug: unmapping init [mem 0xffffffffb7e03000-0xffffffffb7ffffff] [ 3.948448] debug: unmapping init [mem 0xffffffffb6b82000-0xffffffffb6e58fff] [ 3.957072] Write protecting the kernel read-only data: 28672k [ 3.960900] debug: unmapping init [mem 0xffffffffb5203000-0xffffffffb53fffff] [ 3.963504] debug: unmapping init [mem 0xffffffffb5b14000-0xffffffffb5bfffff] [ 3.998722] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.007551] systemd[1]: Detected virtualization kvm. [ 4.009433] systemd[1]: Detected architecture x86-64. [ 4.011461] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.036029] systemd[1]: No hostname configured. [ 4.037773] systemd[1]: Set hostname to . [ 4.039916] random: systemd: uninitialized urandom read (16 bytes read) [ 4.042229] systemd[1]: Initializing machine ID from random generator. [ 4.092397] random: ln: uninitialized urandom read (6 bytes read) [ 4.181561] random: systemd: uninitialized urandom read (16 bytes read) [ 4.184300] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 4.188284] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 4.192110] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Reached target Swap. [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.749681] device-mapper: uevent: version 1.0.3 [ 4.751843] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.470800] virtio_net virtio0 ens2: renamed from eth0 [ 5.473557] random: fast init done [ 5.547421] scsi host0: ata_piix [ 5.611492] scsi host1: ata_piix [ 5.613305] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.615510] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.942836] dracut-initqueue[578]: RTNETLINK answers: File exists [ 10.451105] random: crng init done [ 10.453669] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 12.753698] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 16.638439] printk: systemd: 26 output lines suppressed due to ratelimiting [ 18.129352] SELinux: Disabled at runtime. [ 18.389307] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 18.415253] systemd[1]: Detected virtualization kvm. [ 18.421099] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 21.526218] systemd[1]: initrd-switch-root.service: Succeeded. [ 21.540362] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 21.570399] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 21.589314] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 21.602867] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 21.671918] systemd[1]: Starting Journal Service... Starting Journal Service... [ 21.712066] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Control Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on Process Core Dump Socket. Mounting Huge Pages File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. Mounting POSIX Message Queue File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ 22.132087] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Create list of required st…ce nodes for the current kernel... Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Stopped target Switch Root. Mounting Kernel Debug File System... [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-getty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Apply Kernel Variables... [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 23.830692] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 25.465357] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 25.467136] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 25.974301] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 26.105233] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit) [*** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (11s / no limit)[ 32.500234] Key type dns_resolver registered [ ***] A start job is running for Configur…only root support (11s / no limit)[ 33.185740] NFS: Registering the id_resolver key type [ 33.188822] Key type id_resolver registered [ 33.191407] Key type id_legacy registered [ **] A start job is running for Configur…only root support (12s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg401-server login: [ 61.900222] hrtimer: interrupt took 12234700 ns [ 117.819154] libcfs: loading out-of-tree module taints kernel. [ 117.908710] Key type ._llcrypt registered [ 117.910235] Key type .llcrypt registered [ 118.111060] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_hostid [ 144.994904] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 147.886304] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 147.899575] alg: No test for adler32 (adler32-zlib) [ 149.381507] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [ 151.114257] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 153.039206] Key type lgssc registered [ 155.224427] Lustre: Echo OBD driver; http://www.lustre.org/ [ 177.936338] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 240.359698] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 266.159401] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 266.225654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 267.707880] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 267.785313] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 267.951875] Lustre: lustre-MDT0000: new disk, initializing [ 268.101217] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 268.150154] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 277.672077] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 307.043367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 307.339042] Lustre: 6525:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 307.497450] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 307.513119] Lustre: Skipped 1 previous similar message [ 307.716792] Lustre: lustre-MDT0001: new disk, initializing [ 307.913339] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 308.014029] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 308.031062] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 314.890632] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 321.977439] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 335.761036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 336.285794] Lustre: lustre-OST0000: new disk, initializing [ 336.297475] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 336.308410] Lustre: 8465:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 336.397830] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 342.109062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 342.131437] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 342.269443] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 345.758448] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 365.237943] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 365.459581] Lustre: lustre-OST0001: new disk, initializing [ 365.465403] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 365.475934] Lustre: 9542:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 365.624505] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 371.802278] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 371.830296] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 371.918895] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 374.661866] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 390.082122] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 400.122878] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 410.568511] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing check_logdir /tmp/testlogs/ [ 419.145870] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing yml_node [ 426.862618] Lustre: DEBUG MARKER: Client: 2.17.56.52 [ 431.309833] Lustre: DEBUG MARKER: MDS: 2.17.56.52 [ 435.258771] Lustre: DEBUG MARKER: OSS: 2.17.56.52 [ 436.919550] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 6 12:56:04 EDT 2026 [ 468.158903] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 473.752232] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 497.633917] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 497.643074] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 497.657425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 500.194551] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 500.215477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 500.224440] Lustre: Skipped 1 previous similar message [ 500.245271] Lustre: Skipped 3 previous similar messages [ 503.161454] Lustre: server umount lustre-MDT0000 complete [ 508.080089] LustreError: 6516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786035437 with bad export cookie 4737170125160852343 [ 508.091814] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 508.092430] LustreError: 6516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 510.434636] LustreError: 12177:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 510.436727] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 510.456316] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 510.461729] LustreError: 12177:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 510.506400] Lustre: Skipped 2 previous similar messages [ 513.443726] LustreError: 12177:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 513.447936] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 513.462087] LustreError: 12177:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 513.510471] Lustre: Skipped 2 previous similar messages [ 514.963629] Lustre: server umount lustre-MDT0001 complete [ 520.664123] Lustre: server umount lustre-OST0000 complete [ 526.718602] Lustre: server umount lustre-OST0001 complete [ 536.368831] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_hostid [ 551.680398] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 608.534954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 609.070412] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 609.114941] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 609.192772] Lustre: lustre-MDT0000: new disk, initializing [ 609.264634] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 609.280371] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 615.771647] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 630.014132] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 630.218312] Lustre: 15103:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 630.305250] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 630.311651] Lustre: Skipped 1 previous similar message [ 630.407210] Lustre: lustre-MDT0001: new disk, initializing [ 630.562961] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 630.648053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 630.663612] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 636.722737] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 649.941575] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 656.547560] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 666.825175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 667.536427] Lustre: lustre-OST0000: new disk, initializing [ 667.556574] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 667.573311] Lustre: 17075:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 667.779838] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 668.874413] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 668.890468] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 669.137707] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 675.921645] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 692.326906] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 702.212799] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 702.630157] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 708.090610] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 708.411352] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 715.237736] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 715.241032] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 715.241887] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 715.313150] Lustre: Skipped 1 previous similar message [ 717.925316] Lustre: server umount lustre-OST0000 complete [ 725.473509] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 725.474782] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 725.489103] Lustre: Skipped 1 previous similar message [ 725.500935] Lustre: Skipped 1 previous similar message [ 731.362467] Lustre: server umount lustre-MDT0000 complete [ 734.758243] LustreError: 15962:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 734.787365] LustreError: 15962:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 737.247330] LustreError: 15096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786035666 with bad export cookie 4737170125160853876 [ 737.259026] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 737.263500] LustreError: 15096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 737.970732] Lustre: server umount lustre-MDT0001 complete [ 741.905839] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 13:01:09 (1786035669) [ 750.445053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 751.220790] LustreError: 19359:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 751.406307] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 756.714341] LustreError: 19360:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 756.761527] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 765.927661] LustreError: 19360:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 765.945546] LustreError: 19360:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 767.766105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 774.382336] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 783.861901] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 790.471874] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 799.976800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 808.562832] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 819.112896] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 836.581630] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 836.589478] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 836.607656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 836.624806] Lustre: Skipped 1 previous similar message [ 836.639814] Lustre: Skipped 3 previous similar messages [ 841.335690] Lustre: server umount lustre-OST0000 complete [ 853.093780] Lustre: server umount lustre-MDT0000 complete [ 856.545749] LustreError: 19354:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 856.573350] LustreError: 19354:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 858.624523] LustreError: 19340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786035788 with bad export cookie 4737170125160854751 [ 858.630507] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 858.651599] LustreError: 19340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 859.496975] Lustre: server umount lustre-MDT0001 complete [ 877.968479] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 882.449624] Key type lgssc unregistered [ 882.876235] LNet: 22919:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 882.893967] LNetError: 22919:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 882.959773] LNet: Removed LNI 192.168.204.101@tcp [ 886.011349] Key type .llcrypt unregistered [ 886.017514] Key type ._llcrypt unregistered [ 919.340283] Key type ._llcrypt registered [ 919.342429] Key type .llcrypt registered [ 919.465627] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 920.943146] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 920.975573] alg: No test for adler32 (adler32-zlib) [ 922.094095] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [ 922.450787] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 924.247221] Key type lgssc registered [ 925.640426] Lustre: Echo OBD driver; http://www.lustre.org/ [ 940.876579] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 940.941955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 943.016651] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 950.671413] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 953.315352] LustreError: 24346:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 953.335867] LustreError: 24346:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 958.441619] LustreError: 24347:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 963.247795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 963.567052] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 963.687113] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 970.845675] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 981.886559] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 992.094657] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1001.219481] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1001.792692] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1010.050855] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 1013.061712] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1022.232471] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1049.060146] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1049.061869] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1049.098309] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1049.126814] Lustre: Skipped 1 previous similar message [ 1052.941368] Lustre: server umount lustre-OST0000 complete [ 1059.296503] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1059.300084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1059.308656] Lustre: Skipped 2 previous similar messages [ 1059.326094] Lustre: Skipped 1 previous similar message [ 1064.422019] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1064.428899] Lustre: Skipped 1 previous similar message [ 1065.082591] Lustre: server umount lustre-MDT0000 complete [ 1068.520686] LustreError: 27287:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1068.536491] LustreError: 27287:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1069.746092] LustreError: 24326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786035999 with bad export cookie 7498704530000784390 [ 1069.761859] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1070.446326] Lustre: server umount lustre-MDT0001 complete [ 1086.651307] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 1090.956539] Key type lgssc unregistered [ 1091.466940] LNet: 28331:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1091.485633] LNetError: 28331:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1091.535145] LNet: Removed LNI 192.168.204.101@tcp [ 1093.289561] Key type .llcrypt unregistered [ 1093.294606] Key type ._llcrypt unregistered [ 1138.671224] Key type ._llcrypt registered [ 1138.672853] Key type .llcrypt registered [ 1138.917771] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 1142.579238] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1142.616666] alg: No test for adler32 (adler32-zlib) [ 1144.804974] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [ 1145.666461] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 1147.823593] Key type lgssc registered [ 1150.661400] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1164.715996] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1164.755975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1166.384526] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1172.778335] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 1175.526463] LustreError: 29755:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1175.545503] LustreError: 29755:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1180.644474] LustreError: 29754:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1184.222606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1184.755224] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1191.992025] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 1204.294166] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1214.547662] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1236.115353] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1236.396821] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1248.729830] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 1259.332582] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1261.042361] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1275.360600] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1275.363331] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1275.389924] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1276.385661] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1276.396276] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1276.398270] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1276.424385] Lustre: Skipped 1 previous similar message [ 1277.544986] Lustre: server umount lustre-OST0000 complete [ 1286.626081] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1286.640578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1286.646090] Lustre: Skipped 1 previous similar message [ 1286.663975] Lustre: Skipped 1 previous similar message [ 1291.744497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1296.865450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1296.879942] Lustre: Skipped 2 previous similar messages [ 1297.375149] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1297.837280] Lustre: server umount lustre-MDT0000 complete [ 1301.986729] LustreError: 30960:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1302.028156] LustreError: 30960:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1303.138333] LustreError: 29736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786036232 with bad export cookie 5895937870002133583 [ 1303.147059] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1303.148387] LustreError: 29736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1303.363684] Lustre: server umount lustre-MDT0001 complete [ 1322.358038] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 1325.839164] Key type lgssc unregistered [ 1326.175728] LNet: 33694:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1326.185253] LNetError: 33694:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1326.198835] LNet: Removed LNI 192.168.204.101@tcp [ 1327.330773] Key type .llcrypt unregistered [ 1327.333230] Key type ._llcrypt unregistered [ 1358.059740] Key type ._llcrypt registered [ 1358.063184] Key type .llcrypt registered [ 1358.185436] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 1359.387319] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1359.668957] alg: No test for adler32 (adler32-zlib) [ 1360.796723] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [ 1361.192691] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 1362.895285] Key type lgssc registered [ 1365.012710] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1378.102778] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1378.134578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1379.646606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1385.836532] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 1390.055408] LustreError: 35108:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1390.098305] LustreError: 35108:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1395.172978] LustreError: 35109:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1398.324535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1399.073653] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1405.750625] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 1416.018283] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1423.190700] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1431.827635] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1432.389293] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1441.733644] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 1449.807678] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1451.691606] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1464.291087] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1464.300910] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1464.324440] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1465.314867] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1465.320083] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1465.324085] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1465.344736] Lustre: Skipped 1 previous similar message [ 1469.609856] Lustre: server umount lustre-OST0000 complete [ 1475.553683] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1475.589139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1475.603836] Lustre: Skipped 1 previous similar message [ 1480.672739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1480.682098] Lustre: Skipped 1 previous similar message [ 1485.807235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1485.828377] Lustre: Skipped 1 previous similar message [ 1489.375178] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1489.833818] Lustre: server umount lustre-MDT0000 complete [ 1490.914577] LustreError: 35103:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1490.938360] LustreError: 35103:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1494.720617] LustreError: 35089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786036424 with bad export cookie 2020669124283274707 [ 1494.723780] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1494.739914] LustreError: 35089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1495.437143] Lustre: server umount lustre-MDT0001 complete [ 1511.556729] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 1515.941636] Key type lgssc unregistered [ 1516.296764] LNet: 38648:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1516.310259] LNetError: 38648:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1516.347898] LNet: Removed LNI 192.168.204.101@tcp [ 1517.791121] Key type .llcrypt unregistered [ 1517.796195] Key type ._llcrypt unregistered [ 1550.488689] Key type ._llcrypt registered [ 1550.491975] Key type .llcrypt registered [ 1550.689386] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 1553.805787] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1553.894148] alg: No test for adler32 (adler32-zlib) [ 1555.542284] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [ 1556.119195] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 1557.935132] Key type lgssc registered [ 1560.201801] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1574.789646] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1574.829515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1576.346413] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1583.687258] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 1585.637097] LustreError: 40071:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1585.687033] LustreError: 40071:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1590.763711] LustreError: 40072:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1595.879381] LustreError: 40071:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1596.427190] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1597.199132] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1602.891492] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 1613.660983] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1622.038235] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1632.122354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1632.592509] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1643.224435] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 1645.955905] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1654.109082] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1662.627844] Lustre: lustre-OST0000: Not available for connect from 192.168.204.1@tcp (stopping) [ 1666.537678] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1666.549686] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1666.563097] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1666.575292] Lustre: Skipped 1 previous similar message [ 1666.592979] Lustre: Skipped 1 previous similar message [ 1667.691171] Lustre: lustre-OST0000: Not available for connect from 192.168.204.1@tcp (stopping) [ 1669.105129] Lustre: server umount lustre-OST0000 complete [ 1694.178061] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1694.186939] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1694.204705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1698.277157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1698.282370] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1699.589096] Lustre: server umount lustre-MDT0000 complete [ 1703.395527] LustreError: 40802:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1703.414029] LustreError: 40802:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1704.393273] LustreError: 40052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786036633 with bad export cookie 11478543380231918613 [ 1704.395074] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1704.405991] LustreError: 40052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1711.185597] Lustre: server umount lustre-MDT0001 complete [ 1726.788917] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 1730.863966] Key type lgssc unregistered [ 1731.179581] LNet: 43793:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1731.191410] LNetError: 43793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1731.208881] LNet: Removed LNI 192.168.204.101@tcp [ 1732.440060] Key type .llcrypt unregistered [ 1732.441584] Key type ._llcrypt unregistered [ 1762.436518] Key type ._llcrypt registered [ 1762.440647] Key type .llcrypt registered [ 1762.603403] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 1764.117646] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1764.170293] alg: No test for adler32 (adler32-zlib) [ 1765.466773] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [ 1765.638672] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 1767.265971] Key type lgssc registered [ 1769.972813] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1783.803275] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1783.894921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1785.629260] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1796.067595] LustreError: 45206:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1796.129768] LustreError: 45206:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1796.865679] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 1801.190450] LustreError: 45207:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1806.306823] LustreError: 45206:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1811.434547] LustreError: 45207:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1816.330839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1816.544753] LustreError: 45206:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1816.978710] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1824.200313] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 1836.839609] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1846.478816] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1856.706629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1857.204819] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1868.238562] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 1880.750862] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1881.056672] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1891.305234] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1891.312110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1891.334269] Lustre: Skipped 2 previous similar messages [ 1891.348777] Lustre: Skipped 2 previous similar messages [ 1894.570682] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.1@tcp (stopping) [ 1896.417908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1896.431394] Lustre: Skipped 1 previous similar message [ 1896.918959] Lustre: server umount lustre-MDT0000 complete [ 1899.642598] LustreError: 46306:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1902.462032] LustreError: 45187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786036832 with bad export cookie 17173106236642113608 [ 1902.471420] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1902.482210] LustreError: 45187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1904.787645] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.1@tcp (stopping) [ 1906.657476] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1909.101847] Lustre: server umount lustre-MDT0001 complete [ 1928.380777] Lustre: server umount lustre-OST0000 complete [ 1955.372585] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 1960.108107] Key type lgssc unregistered [ 1960.671312] LNet: 49083:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1960.692152] LNetError: 49083:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1960.719281] LNet: Removed LNI 192.168.204.101@tcp [ 1962.985885] Key type .llcrypt unregistered [ 1962.994453] Key type ._llcrypt unregistered [ 2009.392271] Key type ._llcrypt registered [ 2009.399688] Key type .llcrypt registered [ 2009.514837] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 2011.266934] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2011.509365] alg: No test for adler32 (adler32-zlib) [ 2012.756514] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [ 2012.916250] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 2014.551169] Key type lgssc registered [ 2016.145572] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2028.792335] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2028.880510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2045.089306] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2050.271334] LustreError: 50472:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.101@tcp: failed processing log, type 4: rc = -110 [ 2077.027751] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2087.098889] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 2101.114264] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2266.885472] Lustre: server umount lustre-OST0000 complete [ 2295.722864] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 2300.689713] Key type lgssc unregistered [ 2301.163923] LNet: 52174:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2301.200302] LNetError: 52174:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2302.244747] LNet: Removed LNI 192.168.204.101@tcp [ 2303.691171] Key type .llcrypt unregistered [ 2303.693902] Key type ._llcrypt unregistered [ 2334.278492] Key type ._llcrypt registered [ 2334.282804] Key type .llcrypt registered [ 2334.403213] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 2335.841428] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2335.895474] alg: No test for adler32 (adler32-zlib) [ 2336.965288] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [ 2337.158601] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 2338.903417] Key type lgssc registered [ 2340.519538] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2353.003098] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2353.017361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2354.905056] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2364.449646] LustreError: 53591:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2364.471481] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 2364.485484] LustreError: 53591:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2369.509825] LustreError: 53592:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2374.625470] LustreError: 53591:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2375.452239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2375.961509] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2381.999996] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 2392.673042] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2401.942789] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2411.593923] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2412.027620] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2424.099783] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 2425.494232] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 2436.929056] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2441.184413] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2441.194602] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2441.203158] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2441.224047] Lustre: Skipped 1 previous similar message [ 2446.300161] Lustre: server umount lustre-OST0000 complete [ 2452.402538] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2452.416878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2452.423752] Lustre: Skipped 2 previous similar messages [ 2457.570167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2457.589088] Lustre: Skipped 1 previous similar message [ 2460.641542] LustreError: 53586:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2460.661963] LustreError: 53586:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2460.892781] Lustre: server umount lustre-MDT0000 complete [ 2465.788307] LustreError: 53587:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2465.818053] LustreError: 53587:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2467.250118] LustreError: 53572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786037396 with bad export cookie 3313160509636227102 [ 2467.255221] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2467.272330] LustreError: 53572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2468.080090] Lustre: server umount lustre-MDT0001 complete [ 2488.149824] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 2492.699768] Key type lgssc unregistered [ 2493.155021] LNet: 57154:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2493.166795] LNetError: 57154:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2493.192839] LNet: Removed LNI 192.168.204.101@tcp [ 2494.605161] Key type .llcrypt unregistered [ 2494.609793] Key type ._llcrypt unregistered [ 2524.659378] Key type ._llcrypt registered [ 2524.665421] Key type .llcrypt registered [ 2524.904752] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 2526.475951] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2526.675514] alg: No test for adler32 (adler32-zlib) [ 2527.942435] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [ 2528.322548] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 2529.991143] Key type lgssc registered [ 2532.083068] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2543.126971] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2543.145502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2558.687337] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2563.807552] LustreError: 58543:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.101@tcp: failed processing log, type 4: rc = -110 [ 2590.495299] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2599.918500] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 2609.714963] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2622.267607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2623.744477] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2623.959725] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2630.114161] LustreError: 59583:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2630.189417] LustreError: 59583:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 2634.509864] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 2635.251073] LustreError: 59585:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2635.297507] LustreError: 59585:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2640.367964] LustreError: 59583:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2640.405479] LustreError: 59583:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2645.487358] LustreError: 59585:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2645.554821] LustreError: 59585:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2650.594587] LustreError: 59583:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2650.621465] LustreError: 59583:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2651.598078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2652.047374] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2659.346293] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 2672.310794] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2680.980480] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2685.415322] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2685.422858] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2685.438288] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2688.020575] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2688.031937] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2688.037435] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2690.645861] Lustre: server umount lustre-OST0000 complete [ 2722.785500] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2722.822677] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2722.832203] Lustre: Skipped 2 previous similar messages [ 2726.235614] Lustre: server umount lustre-MDT0000 complete [ 2727.917793] LustreError: 59588:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2727.995978] LustreError: 59588:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2731.659855] LustreError: 61164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786037661 with bad export cookie 10058853858424244545 [ 2731.667806] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2731.683441] LustreError: 61164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2738.821919] Lustre: server umount lustre-MDT0001 complete [ 2759.084483] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 2763.183709] Key type lgssc unregistered [ 2763.717894] LNet: 62323:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2763.726528] LNetError: 62323:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2763.769179] LNet: Removed LNI 192.168.204.101@tcp [ 2765.224861] Key type .llcrypt unregistered [ 2765.226749] Key type ._llcrypt unregistered [ 2798.070745] Key type ._llcrypt registered [ 2798.072991] Key type .llcrypt registered [ 2798.180810] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 2799.492364] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2799.635536] alg: No test for adler32 (adler32-zlib) [ 2800.730210] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [ 2800.953302] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 2802.615144] Key type lgssc registered [ 2804.774110] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2817.125880] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2817.167251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2819.422205] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2826.604448] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 2828.772442] LustreError: 63746:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2828.795335] LustreError: 63746:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2833.892880] LustreError: 63747:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2837.709518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2838.413845] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2845.431414] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 2855.682974] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2864.213642] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2874.520611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2875.003212] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2885.990682] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 2889.350288] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2897.371236] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2919.393256] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2919.417308] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2919.449663] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2920.425694] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2920.453850] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2924.785333] Lustre: server umount lustre-OST0000 complete [ 2930.657039] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2930.657894] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2930.662571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2930.693142] Lustre: Skipped 2 previous similar messages [ 2935.778799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2935.787654] Lustre: Skipped 1 previous similar message [ 2937.339695] Lustre: server umount lustre-MDT0000 complete [ 2940.913940] LustreError: 64821:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2940.935318] LustreError: 64821:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2943.113880] LustreError: 65592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786037872 with bad export cookie 16039731009608434204 [ 2943.115961] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2943.132495] LustreError: 65592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2943.996334] Lustre: server umount lustre-MDT0001 complete [ 2960.285599] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 2965.025528] Key type lgssc unregistered [ 2965.418776] LNet: 67312:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2965.427224] LNetError: 67312:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2965.447318] LNet: Removed LNI 192.168.204.101@tcp [ 2967.264984] Key type .llcrypt unregistered [ 2967.270050] Key type ._llcrypt unregistered [ 3002.531763] Key type ._llcrypt registered [ 3002.534819] Key type .llcrypt registered [ 3002.624165] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 3004.227821] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3004.377070] alg: No test for adler32 (adler32-zlib) [ 3005.393862] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [ 3005.593330] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 3007.287513] Key type lgssc registered [ 3008.448166] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3019.409540] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 13:39:06 (1786037946) [ 3028.262614] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3028.288830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3029.751619] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3035.284554] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 3040.249264] LustreError: 69019:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3040.271725] LustreError: 69019:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3045.353211] LustreError: 69020:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3046.813758] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3047.515513] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3055.269497] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 3066.985288] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3076.720384] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3087.557599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3088.142253] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3096.935241] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 3100.490052] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 3107.265422] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3113.910186] Lustre: Failing over lustre-MDT0000 [ 3113.952464] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3113.967718] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3113.995892] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3114.241870] Lustre: server umount lustre-MDT0000 complete [ 3116.007125] LustreError: 69019:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3116.018680] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3116.060246] Lustre: Skipped 1 previous similar message [ 3119.236039] LustreError: 70864:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3119.270186] LustreError: 70864:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3124.349822] LustreError: 69015:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3124.378200] LustreError: 69015:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3128.316297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3128.613621] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3128.853634] Lustre: *** cfs_fail_loc=135, val=0*** [ 3129.028803] LustreError: 72110:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 3129.047714] LustreError: 72110:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.101@tcp: cfg command failed: rc = -2 [ 3129.065646] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 3129.065646] [ 3129.081679] LustreError: MGC192.168.204.101@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3129.101605] LustreError: 72099:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3129.115461] LustreError: 72099:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3129.129707] LustreError: 72099:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3129.143929] LustreError: 72099:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 3129.187852] Lustre: server umount lustre-MDT0000 complete [ 3129.194756] LustreError: 72099:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3134.577937] LustreError: 69015:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3134.603734] LustreError: 69015:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 3141.856385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3142.349851] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3142.388823] LustreError: 69011:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3143.288140] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3143.460492] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3145.776262] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3148.273940] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3148.393323] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 3148.477785] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 3153.603801] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 3163.618654] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3163.630579] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3163.642859] Lustre: Skipped 1 previous similar message [ 3163.654858] Lustre: Skipped 1 previous similar message [ 3165.780847] Lustre: server umount lustre-OST0000 complete [ 3173.871566] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3173.890990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3173.906155] Lustre: Skipped 1 previous similar message [ 3173.947331] Lustre: Skipped 1 previous similar message [ 3178.816422] Lustre: server umount lustre-MDT0000 complete [ 3178.981955] LustreError: 69020:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3179.010092] LustreError: 69020:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 3184.312542] LustreError: 68999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786038113 with bad export cookie 9209582038853330215 [ 3184.339587] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3185.293947] Lustre: server umount lustre-MDT0001 complete [ 3203.072569] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 3208.298451] Key type lgssc unregistered [ 3209.017575] LNet: 73916:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3209.028595] LNetError: 73916:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3209.053960] LNet: Removed LNI 192.168.204.101@tcp [ 3210.449193] Key type .llcrypt unregistered [ 3210.451778] Key type ._llcrypt unregistered [ 3242.600875] Key type ._llcrypt registered [ 3242.604888] Key type .llcrypt registered [ 3242.783388] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 3244.077903] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3244.274339] alg: No test for adler32 (adler32-zlib) [ 3245.379361] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [ 3245.678822] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 3247.487187] Key type lgssc registered [ 3249.681570] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3262.217567] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3262.271407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3264.405905] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3272.764750] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 3273.701049] LustreError: 75331:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3273.732582] LustreError: 75331:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3278.816419] LustreError: 75330:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3283.944191] LustreError: 75331:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3284.876390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3285.287378] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3291.441672] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 3302.032567] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3310.468399] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3319.024275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3319.548714] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3328.168652] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 3333.929049] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 3337.977344] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3347.065827] Lustre: Failing over lustre-MDT0000 [ 3347.699631] Lustre: server umount lustre-MDT0000 complete [ 3349.492278] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3349.504695] LustreError: 75331:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3349.523232] Lustre: Skipped 1 previous similar message [ 3349.678035] LustreError: 75331:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3354.594299] LustreError: 75330:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3354.623512] LustreError: 75330:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3364.840515] LustreError: 75325:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3364.864567] LustreError: 75325:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 3365.855256] Lustre: 74641:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786038279/real 1786038279] req@0000000033f35021 x1872796765418752/t0(0) o400->MGC192.168.204.101@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786038295 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3365.925742] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3366.181965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3376.484351] Lustre: *** cfs_fail_loc=a05, val=0*** [ 3376.764257] LustreError: 78426:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 3376.773991] LustreError: 78426:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.101@tcp: cfg command failed: rc = -9 [ 3376.788287] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 3376.788287] [ 3376.804442] LustreError: MGC192.168.204.101@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3376.832525] LustreError: 78413:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 3376.853361] LustreError: 78413:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 3376.881421] LustreError: 78413:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3376.896928] LustreError: 78413:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 3376.949937] Lustre: server umount lustre-MDT0000 complete [ 3376.954413] LustreError: 78413:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 3381.219456] LustreError: 75331:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3381.249369] LustreError: 75331:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 14 previous similar messages [ 3387.635290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3387.920250] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3387.956555] LustreError: 75321:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3388.421166] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3388.515443] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3392.625658] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3393.523279] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3393.572336] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 3393.661838] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 3396.065824] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 3406.817158] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3406.832643] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3406.858100] Lustre: Skipped 1 previous similar message [ 3406.869497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3408.174843] Lustre: server umount lustre-OST0000 complete [ 3413.985299] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3414.000954] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3414.005743] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3414.027970] Lustre: Skipped 1 previous similar message [ 3419.108223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3419.114090] Lustre: Skipped 1 previous similar message [ 3424.224648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3424.237731] Lustre: Skipped 1 previous similar message [ 3427.295129] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3427.853386] Lustre: server umount lustre-MDT0000 complete [ 3429.369662] LustreError: 75331:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3429.419646] LustreError: 75331:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 3433.168799] LustreError: 75312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786038362 with bad export cookie 13232717069625390768 [ 3433.183969] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3433.188638] LustreError: 75312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3433.628433] Lustre: server umount lustre-MDT0001 complete [ 3450.363471] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 3456.013339] Key type lgssc unregistered [ 3456.353506] LNet: 80215:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3456.364148] LNetError: 80215:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3457.385719] LNet: Removed LNI 192.168.204.101@tcp [ 3458.974572] Key type .llcrypt unregistered [ 3458.976993] Key type ._llcrypt unregistered [ 3476.851356] Key type ._llcrypt registered [ 3476.855435] Key type .llcrypt registered [ 3477.082427] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 3477.760493] Key type .llcrypt unregistered [ 3477.763817] Key type ._llcrypt unregistered [ 3509.834900] Key type ._llcrypt registered [ 3509.850790] Key type .llcrypt registered [ 3510.076317] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 3511.848599] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3511.903800] alg: No test for adler32 (adler32-zlib) [ 3513.324404] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [ 3513.828643] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 3515.727217] Key type lgssc registered [ 3517.152868] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3528.865454] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3528.894471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3530.331210] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3536.958321] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 3540.463297] LustreError: 82129:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3540.507165] LustreError: 82129:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3545.577364] LustreError: 82130:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3550.696977] LustreError: 82129:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3551.954721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3552.608509] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3558.967923] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 3574.388309] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3583.110363] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3592.865588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3593.443066] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3600.633212] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 3603.198738] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 3614.679474] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3629.551520] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3629.559395] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3629.563356] Lustre: Skipped 1 previous similar message [ 3629.573795] Lustre: Skipped 1 previous similar message [ 3630.626682] Lustre: server umount lustre-OST0000 complete [ 3639.776427] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3639.784396] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3639.800645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3639.817424] Lustre: Skipped 1 previous similar message [ 3639.834018] Lustre: Skipped 1 previous similar message [ 3643.474929] Lustre: server umount lustre-MDT0000 complete [ 3644.899364] LustreError: 82129:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3644.924210] LustreError: 82129:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3648.538523] LustreError: 84730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786038578 with bad export cookie 15141796993547033038 [ 3648.548184] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3648.559953] LustreError: 84730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3649.268870] Lustre: server umount lustre-MDT0001 complete [ 3669.689512] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 3675.075332] Key type lgssc unregistered [ 3675.660668] LNet: 85710:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3675.672694] LNetError: 85710:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3675.700706] LNet: Removed LNI 192.168.204.101@tcp [ 3676.847654] Key type .llcrypt unregistered [ 3676.854635] Key type ._llcrypt unregistered [ 3712.581362] Key type ._llcrypt registered [ 3712.587865] Key type .llcrypt registered [ 3712.789055] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 3714.889668] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3714.938098] alg: No test for adler32 (adler32-zlib) [ 3716.270711] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [ 3716.806420] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 3718.583180] Key type lgssc registered [ 3720.270905] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3730.758535] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3730.787422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3732.325115] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3738.754286] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 3741.673668] LustreError: 87128:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3741.703995] LustreError: 87128:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3746.787982] LustreError: 87127:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3751.924020] LustreError: 87128:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3752.044240] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3752.517940] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3759.198164] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 3772.043016] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3781.210186] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3791.429424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3791.673375] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3800.972421] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 3802.039616] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 3813.883078] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3822.562469] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3822.565174] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3822.589978] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3822.601713] Lustre: Skipped 1 previous similar message [ 3824.696233] Lustre: server umount lustre-OST0000 complete [ 3832.801141] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3832.805578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3832.826708] Lustre: Skipped 2 previous similar messages [ 3836.748299] Lustre: server umount lustre-MDT0000 complete [ 3837.940987] LustreError: 87127:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3837.974422] LustreError: 87127:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3841.727315] LustreError: 87109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786038771 with bad export cookie 9061822291791987026 [ 3841.729783] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3841.740840] LustreError: 87109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3842.504245] Lustre: server umount lustre-MDT0001 complete [ 3856.411715] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 3860.268188] Key type lgssc unregistered [ 3860.716142] LNet: 90668:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3860.752236] LNetError: 90668:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3861.798898] LNet: Removed LNI 192.168.204.101@tcp [ 3862.896090] Key type .llcrypt unregistered [ 3862.901358] Key type ._llcrypt unregistered [ 3891.375428] Key type ._llcrypt registered [ 3891.377305] Key type .llcrypt registered [ 3891.458746] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 3892.883893] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3893.050391] alg: No test for adler32 (adler32-zlib) [ 3894.023170] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [ 3894.205477] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 3895.858790] Key type lgssc registered [ 3897.256253] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3905.621973] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3905.675837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3907.606245] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3913.664075] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 3917.818051] LustreError: 92082:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3917.854142] LustreError: 92082:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3922.915160] LustreError: 92083:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3927.665513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3928.037640] LustreError: 92082:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3928.717962] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3936.871855] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 3948.143195] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3958.024353] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3968.195759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3968.782987] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3977.974364] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 3978.288994] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 3991.679467] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4008.933914] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4008.936574] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4008.938295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4008.988601] Lustre: Skipped 1 previous similar message [ 4013.172644] Lustre: server umount lustre-OST0000 complete [ 4019.174350] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4019.179406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4019.198397] Lustre: Skipped 1 previous similar message [ 4019.207287] Lustre: Skipped 1 previous similar message [ 4024.304288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4024.324254] Lustre: Skipped 2 previous similar messages [ 4025.441819] Lustre: server umount lustre-MDT0000 complete [ 4029.411695] LustreError: 92082:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4029.429943] LustreError: 92082:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4030.765875] LustreError: 92063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786038960 with bad export cookie 14718347663092016696 [ 4030.776684] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4030.799839] LustreError: 92063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4031.601165] Lustre: server umount lustre-MDT0001 complete [ 4051.388634] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 4054.962143] Key type lgssc unregistered [ 4055.531627] LNet: 95668:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4055.542261] LNetError: 95668:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4055.567218] LNet: Removed LNI 192.168.204.101@tcp [ 4056.853555] Key type .llcrypt unregistered [ 4056.856387] Key type ._llcrypt unregistered [ 4088.556335] Key type ._llcrypt registered [ 4088.558370] Key type .llcrypt registered [ 4088.685570] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 4090.573652] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4090.670270] alg: No test for adler32 (adler32-zlib) [ 4092.247540] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [ 4092.768838] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 4094.496213] Key type lgssc registered [ 4095.565816] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4107.256613] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4107.302430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4123.103354] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4128.225049] LustreError: 97056:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.101@tcp: failed processing log, type 4: rc = -110 [ 4155.041864] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4165.346263] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 4179.209981] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4185.949668] Lustre: server umount lustre-OST0000 complete [ 4197.247947] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 13:58:44 (1786039124) [ 4206.747578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4207.476752] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4212.704539] LustreError: 98696:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4212.719370] LustreError: 98696:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4213.389935] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 4216.802304] LustreError: 98697:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4221.927112] LustreError: 98696:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4224.667387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4225.388911] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4231.499652] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 4243.647945] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4251.670413] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4260.057197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4260.710810] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4269.610297] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 4277.105782] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 4280.331272] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4292.577495] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4292.580707] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4292.600171] Lustre: Skipped 1 previous similar message [ 4297.698389] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4297.712209] Lustre: Skipped 2 previous similar messages [ 4298.690187] Lustre: server umount lustre-OST0000 complete [ 4307.424728] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4307.448378] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4307.466591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4310.085671] Lustre: server umount lustre-MDT0000 complete [ 4313.060734] LustreError: 98691:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4313.083573] LustreError: 98691:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4314.913252] LustreError: 98678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786039244 with bad export cookie 5865644780523032566 [ 4314.920191] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4314.925515] LustreError: 98678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4315.338552] Lustre: server umount lustre-MDT0001 complete [ 4329.205690] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 4333.532416] Key type lgssc unregistered [ 4334.044731] LNet: 102407:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4334.067350] LNetError: 102407:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4335.143370] LNet: Removed LNI 192.168.204.101@tcp [ 4336.294809] Key type .llcrypt unregistered [ 4336.301231] Key type ._llcrypt unregistered [ 4395.381295] Key type ._llcrypt registered [ 4395.384347] Key type .llcrypt registered [ 4395.504259] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 4397.117150] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4397.214412] alg: No test for adler32 (adler32-zlib) [ 4398.335125] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [ 4398.704917] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 4400.463176] Key type lgssc registered [ 4402.135364] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4412.361580] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4412.376763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4413.857068] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4421.606731] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 4424.163919] LustreError: 104597:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4424.183638] LustreError: 104597:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4429.284615] LustreError: 104598:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4434.427404] LustreError: 104597:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4436.053950] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4436.653345] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4442.514973] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 4453.940303] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4461.823352] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4473.918269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4474.718504] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4483.904294] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 4484.451676] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 4493.906724] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4507.618503] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4507.632650] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4507.661413] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4509.666792] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4509.667847] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4509.672204] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4511.827359] Lustre: server umount lustre-OST0000 complete [ 4518.368768] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4518.381932] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4518.416108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4518.427664] Lustre: Skipped 1 previous similar message [ 4523.339612] Lustre: server umount lustre-MDT0000 complete [ 4525.038153] LustreError: 104592:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4525.069285] LustreError: 104592:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4530.154109] LustreError: 104597:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4530.184553] LustreError: 104597:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4530.919098] LustreError: 104578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786039460 with bad export cookie 1056666653619943617 [ 4530.921120] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4530.932676] LustreError: 104578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4531.568932] Lustre: server umount lustre-MDT0001 complete [ 4551.568468] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 4555.258231] Key type lgssc unregistered [ 4555.510867] LNet: 108151:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4555.521657] LNetError: 108151:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4555.595257] LNet: Removed LNI 192.168.204.101@tcp [ 4557.227603] Key type .llcrypt unregistered [ 4557.245721] Key type ._llcrypt unregistered [ 4599.967656] Key type ._llcrypt registered [ 4599.969913] Key type .llcrypt registered [ 4600.155476] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 4602.102653] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4602.147522] alg: No test for adler32 (adler32-zlib) [ 4603.444654] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [ 4603.857030] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 4605.583152] Key type lgssc registered [ 4607.609710] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4619.424351] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4619.464390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4635.231574] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4640.351397] LustreError: 109399:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.101@tcp: failed processing log, type 4: rc = -110 [ 4667.167178] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4677.464623] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 4689.918502] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4699.888691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4700.171711] LustreError: MGC192.168.204.101@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 4700.188530] LustreError: 110426:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 4700.197121] LustreError: 110426:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 4700.207626] LustreError: 110426:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 4700.317690] Lustre: server umount lustre-MDT0000 complete [ 4700.322042] LustreError: 110426:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4718.559176] Lustre: 108733:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786039632/real 1786039632] req@0000000023d77ba6 x1872798189358464/t0(0) o400->MGC192.168.204.101@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786039648 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4718.590140] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4718.624263] LustreError: 109411:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 4739.171430] Lustre: server umount lustre-OST0000 complete [ 4753.465114] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_hostid [ 4766.802527] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 4816.393707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4816.831452] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4816.936543] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4817.071146] Lustre: lustre-MDT0000: new disk, initializing [ 4817.177513] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4817.203388] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4822.456539] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 4835.108155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4835.260776] Lustre: 113222:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4835.318303] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4835.322083] Lustre: Skipped 1 previous similar message [ 4835.404075] Lustre: lustre-MDT0001: new disk, initializing [ 4835.491909] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4835.522483] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4835.547104] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4842.250413] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 4854.266877] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4862.304603] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4870.863758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4871.402239] Lustre: lustre-OST0000: new disk, initializing [ 4871.412204] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4871.422317] Lustre: 115192:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4871.595921] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4879.578512] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 4879.998795] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4880.009424] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4880.161519] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4891.973809] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4898.178249] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4898.516580] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4906.493246] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4906.848781] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4911.074277] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4911.083516] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4911.084710] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4911.087567] Lustre: Skipped 1 previous similar message [ 4911.105929] LustreError: Skipped 1 previous similar message [ 4915.912128] Lustre: server umount lustre-OST0000 complete [ 4921.315106] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4921.317381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4921.335674] Lustre: Skipped 1 previous similar message [ 4921.356292] Lustre: Skipped 2 previous similar messages [ 4926.432526] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4927.768214] Lustre: server umount lustre-MDT0000 complete [ 4932.987809] LustreError: 115191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786039862 with bad export cookie 4191482611195312997 [ 4933.008388] LustreError: 115191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4933.011067] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4933.700368] Lustre: server umount lustre-MDT0001 complete [ 4949.253351] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 14:11:16 (1786039876) [ 4956.040547] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 4980.697361] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_hostid [ 4992.664724] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 5046.356552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5046.724460] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5046.760706] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5046.880319] Lustre: lustre-MDT0000: new disk, initializing [ 5047.055022] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5047.083905] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5052.115392] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 5066.439310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5066.621542] Lustre: 120102:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5066.703895] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5066.708580] Lustre: Skipped 1 previous similar message [ 5066.836371] Lustre: lustre-MDT0001: new disk, initializing [ 5066.988522] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5067.046809] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5067.058663] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5073.382911] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 5086.731553] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5095.585752] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5104.812120] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5105.170846] Lustre: lustre-OST0000: new disk, initializing [ 5105.179673] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5105.195123] Lustre: 122070:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5105.357671] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5114.655545] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 5115.052366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5115.069104] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5115.493208] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5133.031238] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5140.587756] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5141.039017] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5149.719763] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5149.963074] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5156.326480] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5156.341431] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5156.357848] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5156.368435] Lustre: Skipped 1 previous similar message [ 5158.764209] Lustre: server umount lustre-OST0000 complete [ 5164.514231] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5164.532939] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5164.546561] Lustre: Skipped 1 previous similar message [ 5164.562270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5164.570148] Lustre: Skipped 1 previous similar message [ 5170.551324] Lustre: server umount lustre-MDT0000 complete [ 5171.683447] LustreError: 120113:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5171.715748] LustreError: 120113:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 5175.327593] LustreError: 120093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786040104 with bad export cookie 4191482611195313879 [ 5175.331851] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5175.343890] LustreError: 120093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5175.826574] Lustre: server umount lustre-MDT0001 complete [ 5187.180495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5187.874792] LustreError: 124291:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5188.043492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5193.190326] LustreError: 124292:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5193.218218] LustreError: 124292:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5193.503502] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 5198.305559] LustreError: 124291:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5203.428565] LustreError: 124292:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5206.146835] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5213.228429] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 5223.922455] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5234.107035] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5244.969322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5253.357982] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 5263.659780] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5272.092396] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 5282.274457] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5282.281894] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5282.292288] Lustre: Skipped 1 previous similar message [ 5282.299544] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5282.304305] Lustre: Skipped 2 previous similar messages [ 5285.466368] Lustre: server umount lustre-OST0000 complete [ 5293.537872] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5293.546686] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5293.546957] Lustre: Skipped 2 previous similar messages [ 5293.569192] LustreError: Skipped 1 previous similar message [ 5297.770195] Lustre: server umount lustre-MDT0000 complete [ 5298.657274] LustreError: 124287:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5298.675652] LustreError: 124287:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5303.132902] LustreError: 124271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786040232 with bad export cookie 4191482611195314761 [ 5303.168756] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5303.693937] Lustre: server umount lustre-MDT0001 complete [ 5321.046300] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 5324.285897] Key type lgssc unregistered [ 5325.103523] LNet: 127929:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5325.118456] LNetError: 127929:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5325.160369] LNet: Removed LNI 192.168.204.101@tcp [ 5326.887473] Key type .llcrypt unregistered [ 5326.891667] Key type ._llcrypt unregistered [ 5351.017959] Key type ._llcrypt registered [ 5351.021319] Key type .llcrypt registered [ 5351.118285] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_hostid [ 5373.606893] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 5376.961506] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5377.036587] alg: No test for adler32 (adler32-zlib) [ 5378.240633] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [ 5378.852237] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 5380.695145] Key type lgssc registered [ 5382.764669] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5437.076139] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5437.121882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5438.696814] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5438.751582] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5438.875826] Lustre: lustre-MDT0000: new disk, initializing [ 5438.992413] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5439.022422] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5445.485774] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 5459.310069] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5459.548109] Lustre: 131088:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5459.654691] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5459.662961] Lustre: Skipped 1 previous similar message [ 5459.801809] Lustre: lustre-MDT0001: new disk, initializing [ 5459.971554] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5460.103341] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5460.124179] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5465.913812] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 5477.946721] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5485.185105] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5497.829593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5498.312461] Lustre: lustre-OST0000: new disk, initializing [ 5498.328831] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5498.338180] Lustre: 133057:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5498.467392] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5505.678174] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5505.688378] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5505.873472] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5506.881186] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 5520.028906] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5527.272971] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5527.496679] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5533.684169] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5533.925897] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5536.737424] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5536.757026] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5536.770223] Lustre: Skipped 1 previous similar message [ 5536.797108] Lustre: Skipped 1 previous similar message [ 5541.863966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5541.871443] Lustre: Skipped 1 previous similar message [ 5543.098070] Lustre: server umount lustre-OST0000 complete [ 5552.096352] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5552.103996] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5552.117842] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5552.131763] Lustre: Skipped 1 previous similar message [ 5552.156476] Lustre: Skipped 1 previous similar message [ 5557.222315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5557.229215] Lustre: Skipped 1 previous similar message [ 5562.337450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5562.337870] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 4. Is it stuck? [ 5562.345712] Lustre: Skipped 1 previous similar message [ 5564.622647] Lustre: server umount lustre-MDT0000 complete [ 5568.973943] LustreError: 131080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786040498 with bad export cookie 8540773936319575648 [ 5568.974898] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5568.999047] LustreError: 131080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5569.330984] Lustre: server umount lustre-MDT0001 complete [ 5581.353186] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 14:21:48 (1786040508) [ 5590.577658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5591.419319] LustreError: 135475:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5591.447364] LustreError: 135475:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5591.542769] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5596.650230] LustreError: 135476:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5596.964985] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 5601.771912] LustreError: 135475:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5606.887621] LustreError: 135476:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5607.869567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5608.398211] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5614.096268] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 5624.352256] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5631.275537] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5638.117555] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5638.133178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5638.136241] Lustre: Skipped 1 previous similar message [ 5638.160331] Lustre: Skipped 1 previous similar message [ 5640.841281] Lustre: server umount lustre-MDT0000 complete [ 5643.233895] LustreError: 135472:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5643.257474] LustreError: 135472:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5646.111780] LustreError: 135457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786040575 with bad export cookie 8540773936319576530 [ 5646.124723] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5646.134065] LustreError: 135457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5646.934109] Lustre: server umount lustre-MDT0001 complete [ 5658.725493] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 14:23:05 (1786040585) [ 5667.600151] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5683.295627] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5688.415889] LustreError: 137895:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.101@tcp: failed processing log, type 4: rc = -110 [ 5714.143643] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5726.175886] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 5739.323447] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5744.058651] Lustre: server umount lustre-OST0000 complete [ 5756.295707] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 14:24:43 (1786040683) [ 5765.543896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5766.224067] LustreError: 139345:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5766.385362] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5772.803806] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 5785.994480] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5786.083722] LustreError: 139346:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5786.124194] LustreError: 139346:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5794.370463] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 5805.664812] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5813.918617] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5823.733661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5824.086167] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5824.096182] Lustre: Skipped 1 previous similar message [ 5832.162924] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 5848.164886] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5862.369569] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5862.399681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5862.405607] Lustre: Skipped 2 previous similar messages [ 5862.428617] Lustre: Skipped 2 previous similar messages [ 5866.653863] Lustre: server umount lustre-MDT0000 complete [ 5867.514343] LustreError: 140405:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5867.550732] LustreError: 140405:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5872.729840] LustreError: 139326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786040802 with bad export cookie 8540773936319577195 [ 5872.759313] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5873.053351] Lustre: server umount lustre-MDT0001 complete [ 5887.653249] Lustre: server umount lustre-OST0000 complete [ 5897.668219] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 14:27:04 (1786040824) [ 5907.324706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5908.104897] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5913.556518] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 5923.837562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5930.694147] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 5941.240852] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5951.572353] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5959.629549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5968.982484] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 5978.447077] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 5981.615867] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5990.193492] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5990.709282] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5998.178848] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5998.553863] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6004.194311] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6004.199361] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6004.201336] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6004.234457] Lustre: Skipped 1 previous similar message [ 6007.480637] Lustre: server umount lustre-OST0000 complete [ 6014.432637] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6018.927111] Lustre: server umount lustre-MDT0000 complete [ 6019.552994] LustreError: 143004:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6019.585190] LustreError: 143004:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 6023.694503] LustreError: 142984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786040953 with bad export cookie 8540773936319578266 [ 6023.696122] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6023.705234] LustreError: 142984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 6024.377591] Lustre: server umount lustre-MDT0001 complete [ 6036.049724] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 14:29:22 (1786040962) [ 6050.019168] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6066.080081] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6071.201695] LustreError: 147169:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.101@tcp: failed processing log, type 4: rc = -110 [ 6096.927667] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6096.942946] Lustre: Skipped 2 previous similar messages [ 6106.842941] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 6120.456772] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6133.418732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6134.784756] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 6142.981204] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 6150.140096] LustreError: 147194:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6150.204557] LustreError: 147194:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6157.185414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6166.397616] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 6180.175865] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6188.699828] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6195.171192] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6195.498705] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6202.328119] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6202.546381] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6208.995157] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6209.004070] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6209.018069] Lustre: Skipped 1 previous similar message [ 6209.025684] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6209.028256] Lustre: Skipped 3 previous similar messages [ 6219.232056] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6220.090701] Lustre: server umount lustre-MDT0000 complete [ 6225.260445] LustreError: 147175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786041154 with bad export cookie 8540773936319579288 [ 6225.268404] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6225.282129] LustreError: 147175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6226.166137] Lustre: server umount lustre-MDT0001 complete [ 6242.071589] Lustre: server umount lustre-OST0000 complete [ 6253.016041] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 14:33:00 (1786041180) [ 6262.639868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6278.239632] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6283.359622] LustreError: 151375:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.101@tcp: failed processing log, type 4: rc = -110 [ 6318.367817] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 6327.485022] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6338.675533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6339.909854] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 6345.908909] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 6358.352775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6358.932631] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6358.943326] Lustre: Skipped 4 previous similar messages [ 6365.196498] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 6375.955432] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6383.796656] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6393.052270] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6393.453415] Lustre: lustre-OST0001: new disk, initializing [ 6393.457443] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6393.478625] Lustre: 154251:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6400.130849] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6400.156921] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6400.376521] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6403.052930] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 6416.815701] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6423.270492] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6423.434900] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6429.567721] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6429.847545] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6435.809269] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6435.818059] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6435.837650] Lustre: Skipped 2 previous similar messages [ 6435.849803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6435.859707] Lustre: Skipped 6 previous similar messages [ 6438.883144] Lustre: server umount lustre-OST0000 complete [ 6441.441828] LustreError: 151391:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6441.459393] LustreError: 151391:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 17 previous similar messages [ 6446.560670] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6450.094731] Lustre: server umount lustre-OST0001 complete [ 6456.290119] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6456.296566] LustreError: Skipped 1 previous similar message [ 6462.062913] Lustre: server umount lustre-MDT0000 complete [ 6467.457152] LustreError: 151382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786041397 with bad export cookie 8540773936319580240 [ 6467.464341] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6468.112902] Lustre: server umount lustre-MDT0001 complete [ 6497.932832] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 14:37:05 (1786041425) [ 6500.618574] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 6502.735941] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 6506.268352] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 14:37:12 (1786041432) [ 6516.547605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6517.118843] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6517.192581] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6522.719490] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 6535.529222] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6535.790279] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6535.872010] Lustre: 157814:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6541.895978] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 6554.985988] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6566.439591] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6575.535965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6575.940840] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6584.256636] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 6586.560433] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 6594.851506] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6600.750592] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6600.933254] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6606.399317] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6606.582494] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6612.448139] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6612.452033] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6612.474371] Lustre: Skipped 6 previous similar messages [ 6623.199383] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6623.566684] Lustre: server umount lustre-OST0000 complete [ 6651.950308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6653.439115] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6653.463116] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6653.463591] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6661.565680] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 6672.653850] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6683.653920] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6684.236131] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6690.451786] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6690.876618] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6698.359763] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8cf845cbe800.ost_server_uuid 50 [ 6700.329274] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8cf845cbe800.ost_server_uuid in IDLE state after 0 sec [ 6700.567279] LustreError: 157822:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.204.1@tcp) returned error from blocking AST (req@00000000b98b96cf x1872799002314112 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000bdf69f96/0x7686eca4e53e51aa lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.1@tcp remote: 0x306517745e3952b8 expref: 5 pid: 157821 timeout: 6800 lvb_type: 0 lru_score: 0 lru_type: 0 [ 6700.615794] LustreError: lustre-MDT0000: A client on nid 192.168.204.1@tcp was evicted due to a lock blocking callback time out: rc -107 [ 6700.625112] LustreError: 157812:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.204.1@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000bdf69f96/0x7686eca4e53e51aa lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.1@tcp remote: 0x306517745e3952b8 expref: 6 pid: 157821 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 6709.743098] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6709.755783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6709.763257] Lustre: Skipped 13 previous similar messages [ 6714.038623] Lustre: server umount lustre-OST0000 complete [ 6726.971093] Lustre: server umount lustre-MDT0000 complete [ 6732.390951] LustreError: 157807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786041661 with bad export cookie 8540773936319581423 [ 6732.396034] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6732.406887] LustreError: 157807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6740.049351] Lustre: server umount lustre-MDT0001 complete [ 6755.024370] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 6758.496280] Key type lgssc unregistered [ 6758.883677] LNet: 163949:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6758.895139] LNetError: 163949:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6758.919227] LNet: Removed LNI 192.168.204.101@tcp [ 6759.946924] Key type .llcrypt unregistered [ 6759.951954] Key type ._llcrypt unregistered [ 6792.907819] Key type ._llcrypt registered [ 6792.915358] Key type .llcrypt registered [ 6793.091550] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 6794.840634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6795.030099] alg: No test for adler32 (adler32-zlib) [ 6796.247487] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [ 6796.719743] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 6798.537461] Key type lgssc registered [ 6800.322323] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6811.302484] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6811.333061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6812.979736] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6817.932458] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 6823.401341] LustreError: 165366:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6823.419719] LustreError: 165366:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6828.517373] LustreError: 165367:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6828.635115] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6829.274465] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6835.198878] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 6844.830246] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6853.520228] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6863.132203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6863.616489] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6871.663458] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 6880.162889] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6883.056927] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 6890.835233] Lustre: Failing over lustre-MDT0000 [ 6891.178932] LustreError: 165363:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6891.468251] Lustre: server umount lustre-MDT0000 complete [ 6893.548536] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6893.555116] LustreError: 166094:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6893.561760] Lustre: Skipped 2 previous similar messages [ 6893.575264] LustreError: 166094:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6898.657133] LustreError: 165367:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6898.673513] LustreError: 165367:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6903.999995] LustreError: 167213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.1@tcp arrived at 1786041833 with bad export cookie 5193470788213933467 [ 6904.019070] LustreError: 167213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6908.898793] LustreError: 166094:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6908.937731] LustreError: 166094:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6909.919865] Lustre: 164676:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786041823/real 1786041823] req@00000000be541214 x1872800488912000/t0(0) o400->MGC192.168.204.101@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786041839 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6909.951916] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6915.198815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6920.175737] LustreError: 164674:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000096db78d8 x1872800488917376/t0(0) o250->MGC192.168.204.101@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6920.443105] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6920.506941] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6925.818620] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6925.820373] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6928.013819] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 6931.155057] Lustre: lustre-MDT0000: Denying connection for new client d7d870c6-1043-495c-9825-84aa3e569c71 (at 192.168.204.1@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:04 [ 6936.177191] Lustre: lustre-MDT0000: Denying connection for new client d7d870c6-1043-495c-9825-84aa3e569c71 (at 192.168.204.1@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 6940.641739] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6940.656036] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6940.687513] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6946.290583] LustreError: 167191:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6946.312517] LustreError: 167191:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 6947.042689] Lustre: server umount lustre-OST0000 complete [ 6952.131616] LustreError: 169105:0:(ldlm_lib.c:3004:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 6952.149700] Lustre: 168454:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 6952.163963] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 6952.206084] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 6952.230421] LustreError: 168454:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000b7e9abfc x1872800488933632/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6952.276798] LustreError: 168454:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 6952.302163] LustreError: 168454:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 6952.323105] Lustre: lustre-MDT0000: Recovery over after 0:27, of 2 clients 0 recovered and 2 were evicted. [ 6953.162392] Lustre: server umount lustre-MDT0000 complete [ 6956.512586] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6956.534172] Lustre: Skipped 1 previous similar message [ 6960.417149] LustreError: 165347:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786041889 with bad export cookie 5193470788213933789 [ 6960.440698] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6960.443405] LustreError: 165347:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6967.083983] Lustre: server umount lustre-MDT0001 complete [ 6987.253378] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 6990.914899] Key type lgssc unregistered [ 6991.302655] LNet: 169883:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6991.308216] LNetError: 169883:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6991.320882] LNet: Removed LNI 192.168.204.101@tcp [ 6992.759750] Key type .llcrypt unregistered [ 6992.764779] Key type ._llcrypt unregistered [ 7030.131680] Key type ._llcrypt registered [ 7030.135882] Key type .llcrypt registered [ 7030.277807] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 7032.240229] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7032.280346] alg: No test for adler32 (adler32-zlib) [ 7033.407139] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [ 7033.648104] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 7035.375578] Key type lgssc registered [ 7037.092907] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7047.092661] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7047.143882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7049.171127] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7055.161234] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 7059.429544] LustreError: 171298:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7059.461103] LustreError: 171298:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7064.560062] LustreError: 171299:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7068.647579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7069.103088] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7077.124419] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 7087.737961] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7095.196366] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7105.089994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7105.622699] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7115.223592] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 7119.050792] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 7125.937324] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7139.815749] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7139.817110] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7139.839531] LustreError: Skipped 1 previous similar message [ 7139.840863] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7139.877139] Lustre: Skipped 1 previous similar message [ 7141.421881] Lustre: server umount lustre-OST0000 complete [ 7150.053234] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7150.063479] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7150.085988] Lustre: Skipped 1 previous similar message [ 7150.110426] Lustre: Skipped 2 previous similar messages [ 7152.940102] Lustre: server umount lustre-MDT0000 complete [ 7155.168364] LustreError: 171298:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7155.182856] LustreError: 171298:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7158.793157] LustreError: 171280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786042088 with bad export cookie 7813686566144587718 [ 7158.811747] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7159.138277] Lustre: server umount lustre-MDT0001 complete [ 7178.697293] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 7184.601369] Key type lgssc unregistered [ 7185.248753] LNet: 174860:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7185.261302] LNetError: 174860:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7185.335230] LNet: Removed LNI 192.168.204.101@tcp [ 7187.421760] Key type .llcrypt unregistered [ 7187.428709] Key type ._llcrypt unregistered [ 7238.640631] Key type ._llcrypt registered [ 7238.652652] Key type .llcrypt registered [ 7239.037419] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 7240.993589] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7241.021049] alg: No test for adler32 (adler32-zlib) [ 7242.650480] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [ 7242.998309] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 7244.815225] Key type lgssc registered [ 7246.669570] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7260.052578] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7260.137033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7262.297269] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7268.552865] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 7272.448603] LustreError: 176562:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7272.476762] LustreError: 176562:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7277.538474] LustreError: 176561:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7279.909668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7280.440980] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7286.624442] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 7296.682607] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7305.598783] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7317.364269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7317.705319] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7327.463719] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 7333.112733] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 7338.750944] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7348.710074] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7348.721387] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7348.734980] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7353.828743] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7353.838029] Lustre: Skipped 2 previous similar messages [ 7354.759557] Lustre: server umount lustre-OST0000 complete [ 7362.558160] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7362.581369] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7362.598982] Lustre: Skipped 1 previous similar message [ 7362.620430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7364.085110] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7366.897887] Lustre: server umount lustre-MDT0000 complete [ 7369.184416] LustreError: 176562:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7369.202413] LustreError: 176562:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7372.584603] LustreError: 176543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786042302 with bad export cookie 249349996593666072 [ 7372.594980] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7372.595871] LustreError: 176543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7373.304915] Lustre: server umount lustre-MDT0001 complete [ 7390.154595] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 7394.211777] Key type lgssc unregistered [ 7394.753535] LNet: 180105:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7394.768680] LNetError: 180105:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7394.804667] LNet: Removed LNI 192.168.204.101@tcp [ 7396.245186] Key type .llcrypt unregistered [ 7396.249180] Key type ._llcrypt unregistered [ 7439.686297] Key type ._llcrypt registered [ 7439.696442] Key type .llcrypt registered [ 7440.051223] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 7442.065304] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7442.167243] alg: No test for adler32 (adler32-zlib) [ 7443.499884] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [ 7443.768878] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 7445.463166] Key type lgssc registered [ 7447.800651] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7461.840699] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7461.883454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7463.441754] Lustre: *** cfs_fail_loc=135, val=0*** [ 7463.581153] LustreError: 181631:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 7463.592332] LustreError: 181631:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.101@tcp: cfg command failed: rc = -2 [ 7463.603591] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 7463.603591] [ 7463.616224] LustreError: MGC192.168.204.101@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 7463.631342] LustreError: 181592:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 7463.640578] LustreError: 181592:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 7463.653762] LustreError: 181592:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 7463.664957] LustreError: 181592:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 7463.791801] Lustre: server umount lustre-MDT0000 complete [ 7463.797765] LustreError: 181592:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 7473.125330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7473.563319] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7478.739544] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 7483.874642] LustreError: 182002:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7483.889345] LustreError: 182002:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7488.998329] LustreError: 182003:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7489.892514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7490.713626] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7497.000347] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 7507.275587] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7514.872724] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7520.739026] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7520.749444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7520.755483] Lustre: Skipped 1 previous similar message [ 7520.774414] Lustre: Skipped 1 previous similar message [ 7524.295476] Lustre: server umount lustre-MDT0000 complete [ 7525.859741] LustreError: 182002:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7525.896428] LustreError: 182002:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7529.509025] LustreError: 181985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786042459 with bad export cookie 13446857963723006002 [ 7529.524258] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7529.844275] Lustre: server umount lustre-MDT0001 complete [ 7547.410186] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 7553.646606] Key type lgssc unregistered [ 7554.122387] LNet: 184330:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7554.140351] LNetError: 184330:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7554.172596] LNet: Removed LNI 192.168.204.101@tcp [ 7555.752834] Key type .llcrypt unregistered [ 7555.764360] Key type ._llcrypt unregistered [ 7590.840481] Key type ._llcrypt registered [ 7590.842757] Key type .llcrypt registered [ 7591.030629] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 7591.398302] Key type .llcrypt unregistered [ 7591.400146] Key type ._llcrypt unregistered [ 7616.345911] Key type ._llcrypt registered [ 7616.348406] Key type .llcrypt registered [ 7616.569751] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 7619.218280] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7619.286214] alg: No test for adler32 (adler32-zlib) [ 7620.451926] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [ 7620.929726] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 7622.807723] Key type lgssc registered [ 7624.210149] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7633.399062] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7633.410791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7648.928551] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7654.047336] LustreError: 186470:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.101@tcp: failed processing log, type 4: rc = -110 [ 7680.739394] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7689.297296] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 7698.384486] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7706.604586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7707.159748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7707.226579] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 7712.228808] LustreError: 186495:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7712.248208] LustreError: 186495:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7712.912221] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 7716.326522] LustreError: 187508:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7716.354992] LustreError: 187508:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7721.483558] LustreError: 187509:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7721.523551] LustreError: 187509:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7726.570179] LustreError: 186496:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7726.608492] LustreError: 186496:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7726.678548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7727.294126] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7735.547044] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 7748.066571] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7758.245641] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7766.316445] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 7783.909843] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7783.912371] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7783.923769] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7783.934587] Lustre: Skipped 1 previous similar message [ 7783.955703] Lustre: Skipped 1 previous similar message [ 7786.824516] Lustre: server umount lustre-OST0000 complete [ 7793.635499] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7793.645710] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7793.661993] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7799.269751] LustreError: 187512:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7799.300021] LustreError: 187512:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7799.488165] Lustre: server umount lustre-MDT0000 complete [ 7804.565944] LustreError: 186478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786042734 with bad export cookie 13902530592030611884 [ 7804.569156] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7804.588577] LustreError: 186478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7805.168605] Lustre: server umount lustre-MDT0001 complete [ 7821.709585] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 7826.094272] Key type lgssc unregistered [ 7826.549407] LNet: 190398:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7826.563817] LNetError: 190398:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7826.588898] LNet: Removed LNI 192.168.204.101@tcp [ 7827.657423] Key type .llcrypt unregistered [ 7827.658860] Key type ._llcrypt unregistered [ 7867.663320] Key type ._llcrypt registered [ 7867.670935] Key type .llcrypt registered [ 7867.942714] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 7869.678093] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7869.810141] alg: No test for adler32 (adler32-zlib) [ 7871.236479] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [ 7871.698832] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 7873.511562] Key type lgssc registered [ 7874.674352] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7885.006973] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7885.061461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7886.720288] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7895.156172] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 7897.090147] LustreError: 191833:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7897.146532] LustreError: 191833:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7902.182127] LustreError: 191834:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7907.299478] LustreError: 191833:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7911.458475] LustreError: 191834:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7913.599478] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7914.638226] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7923.672329] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 7935.771254] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7943.786431] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7951.795458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7952.199448] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7957.364472] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 7961.243080] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 7972.371608] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7979.299476] Lustre: Failing over lustre-MDT0000 [ 7979.906741] Lustre: server umount lustre-MDT0000 complete [ 7980.219299] LustreError: 191828:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7981.028069] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7981.030765] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7981.117468] Lustre: Skipped 2 previous similar messages [ 7990.393869] LustreError: 191830:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7990.428267] LustreError: 191830:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 7997.407367] Lustre: 191144:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786042910/real 1786042910] req@00000000dfd3c5e5 x1872801616132992/t0(0) o400->MGC192.168.204.101@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786042926 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7997.450203] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8002.596197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8006.766554] LustreError: 191828:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8006.789504] LustreError: 191828:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 16 previous similar messages [ 8007.648020] LustreError: 191142:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000037acd577 x1872801616138752/t0(0) o250->MGC192.168.204.101@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8008.124073] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8008.225498] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8011.918079] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8013.304843] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8013.352895] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 8013.486767] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 8014.834160] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 8022.166153] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 8025.585573] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 8042.469946] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8042.495202] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8042.534636] LustreError: 191142:0:(events.c:296:request_in_callback()) event type 2, status -110, service ost [ 8042.621150] Lustre: server umount lustre-OST0000 complete [ 8048.042557] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8048.045585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8048.064830] Lustre: Skipped 1 previous similar message [ 8048.094861] Lustre: Skipped 1 previous similar message [ 8053.153271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8053.164698] Lustre: Skipped 1 previous similar message [ 8058.344148] LustreError: 191830:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8058.381518] LustreError: 191830:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 8058.572707] Lustre: server umount lustre-MDT0000 complete [ 8064.375791] LustreError: 195538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786042993 with bad export cookie 13058254484035168922 [ 8064.386638] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8064.390500] LustreError: 195538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8065.037513] Lustre: server umount lustre-MDT0001 complete [ 8089.274288] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 8095.665301] Key type lgssc unregistered [ 8096.218525] LNet: 196649:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8096.233864] LNetError: 196649:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8096.253948] LNet: Removed LNI 192.168.204.101@tcp [ 8097.607124] Key type .llcrypt unregistered [ 8097.610529] Key type ._llcrypt unregistered [ 8127.452358] Key type ._llcrypt registered [ 8127.455228] Key type .llcrypt registered [ 8127.548383] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 8128.885568] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8128.925762] alg: No test for adler32 (adler32-zlib) [ 8129.960155] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [ 8130.289344] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 8131.944087] Key type lgssc registered [ 8132.996040] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8143.143984] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8143.172414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8144.911120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8150.883546] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 8155.114539] LustreError: 198072:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8155.139620] LustreError: 198072:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8160.230358] LustreError: 198073:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8161.553293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8162.143487] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8167.950100] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 8178.303979] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8187.008976] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8196.885856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8197.438207] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8204.630794] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 8214.425540] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8215.930296] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 8223.581088] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 8230.804544] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 8249.339389] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 8268.258065] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8268.273886] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8268.309928] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8272.381450] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8272.386254] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8272.402720] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8272.463928] Lustre: Skipped 1 previous similar message [ 8273.903130] Lustre: server umount lustre-OST0000 complete [ 8282.597395] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8282.625772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8282.634584] Lustre: Skipped 1 previous similar message [ 8287.286480] Lustre: server umount lustre-MDT0000 complete [ 8287.730446] LustreError: 198801:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8287.767290] LustreError: 198801:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8292.842248] LustreError: 198069:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8292.860824] LustreError: 198069:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8293.884316] LustreError: 198053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786043223 with bad export cookie 791934858274171157 [ 8293.902017] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8293.906868] LustreError: 198053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8294.831939] Lustre: server umount lustre-MDT0001 complete [ 8311.569908] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 8315.870703] Key type lgssc unregistered [ 8316.340540] LNet: 201956:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8316.355461] LNetError: 201956:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8316.373936] LNet: Removed LNI 192.168.204.101@tcp [ 8317.608710] Key type .llcrypt unregistered [ 8317.611814] Key type ._llcrypt unregistered [ 8345.381378] Key type ._llcrypt registered [ 8345.385818] Key type .llcrypt registered [ 8345.522050] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 8346.779527] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8346.998717] alg: No test for adler32 (adler32-zlib) [ 8348.744219] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [ 8349.373940] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 8351.047117] Key type lgssc registered [ 8353.185807] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8363.207953] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8363.222886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8364.627154] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8372.282799] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 8374.756425] LustreError: 203223:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8374.787623] LustreError: 203223:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8379.874480] LustreError: 203224:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8385.016221] LustreError: 203223:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8385.279208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8386.449697] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8393.176350] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 8404.035990] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8412.349145] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8422.714158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8423.270823] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8433.049319] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 8434.603414] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 8442.965429] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8450.834343] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 8463.794120] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 8472.469464] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 8485.021138] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 8501.217375] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8501.265477] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8501.278022] Lustre: Skipped 1 previous similar message [ 8506.359244] LustreError: 205047:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8506.388827] LustreError: 205047:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8506.595945] Lustre: server umount lustre-OST0000 complete [ 8514.528504] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8514.541139] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8514.576045] Lustre: Skipped 1 previous similar message [ 8514.583253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8516.583943] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8516.598075] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8516.635398] Lustre: Skipped 1 previous similar message [ 8521.704810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8521.720135] Lustre: Skipped 1 previous similar message [ 8526.303444] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8526.432885] Lustre: server umount lustre-MDT0000 complete [ 8526.820983] LustreError: 203218:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8526.835971] LustreError: 203218:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8530.970057] LustreError: 206903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786043460 with bad export cookie 12310781675785677150 [ 8530.982993] LustreError: 206903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8530.990867] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8532.186773] Lustre: server umount lustre-MDT0001 complete [ 8548.967113] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 8552.709983] Key type lgssc unregistered [ 8553.122678] LNet: 208208:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8553.133579] LNetError: 208208:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8553.145959] LNet: Removed LNI 192.168.204.101@tcp [ 8554.260565] Key type .llcrypt unregistered [ 8554.263153] Key type ._llcrypt unregistered [ 8583.730829] Key type ._llcrypt registered [ 8583.732780] Key type .llcrypt registered [ 8584.018701] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 8586.384444] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8586.423557] alg: No test for adler32 (adler32-zlib) [ 8587.620640] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [ 8587.896216] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 8589.679194] Key type lgssc registered [ 8592.364643] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8603.519984] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8603.550084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8605.408023] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8612.895962] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 8614.882052] LustreError: 209626:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8614.910246] LustreError: 209626:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8620.002739] LustreError: 209627:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8624.892033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8625.133131] LustreError: 209626:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8625.556577] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8636.971483] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 8654.650090] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8662.810298] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8672.551885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8673.096390] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8675.166779] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 8683.606666] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 8695.886812] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8720.352243] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8720.358604] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8720.365516] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8720.391045] Lustre: Skipped 1 previous similar message [ 8724.137964] Lustre: server umount lustre-OST0000 complete [ 8730.596663] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8730.599988] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8730.633752] Lustre: Skipped 1 previous similar message [ 8730.654852] Lustre: Skipped 2 previous similar messages [ 8735.749730] LustreError: 210432:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8735.783309] LustreError: 210432:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8736.330086] Lustre: server umount lustre-MDT0000 complete [ 8740.841033] LustreError: 209627:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8740.870969] LustreError: 209627:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8741.704327] LustreError: 209606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786043671 with bad export cookie 17255211611281534148 [ 8741.718060] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8741.719359] LustreError: 209606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8742.410531] Lustre: server umount lustre-MDT0001 complete [ 8764.567477] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 8770.133113] Key type lgssc unregistered [ 8770.382916] LNet: 213342:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8770.395056] LNetError: 213342:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8771.428684] LNet: Removed LNI 192.168.204.101@tcp [ 8772.696839] Key type .llcrypt unregistered [ 8772.701920] Key type ._llcrypt unregistered [ 8801.744358] Key type ._llcrypt registered [ 8801.753636] Key type .llcrypt registered [ 8801.987578] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 8804.181203] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8804.216457] alg: No test for adler32 (adler32-zlib) [ 8805.536420] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [ 8805.981298] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 8807.847215] Key type lgssc registered [ 8809.503715] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8819.287705] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8819.322591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8821.058341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8826.784912] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 8831.461923] LustreError: 214610:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8831.484136] LustreError: 214610:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8836.610116] LustreError: 214611:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8838.852529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8839.375436] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8847.887373] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 8860.496823] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8869.176017] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8880.881635] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8881.565550] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8890.313758] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 8891.844524] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 8901.758494] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8922.596447] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8922.610906] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8922.620451] Lustre: Skipped 1 previous similar message [ 8922.642411] Lustre: Skipped 1 previous similar message [ 8927.491544] Lustre: server umount lustre-OST0000 complete [ 8936.416697] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8936.425086] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8936.449262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8937.952491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8937.953821] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8943.073531] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8943.085969] Lustre: Skipped 2 previous similar messages [ 8947.167222] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8947.727603] Lustre: server umount lustre-MDT0000 complete [ 8949.227166] LustreError: 214611:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8949.257204] LustreError: 214611:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8953.388270] LustreError: 214593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786043882 with bad export cookie 15629226155977993181 [ 8953.397095] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8953.411267] LustreError: 214593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8954.058060] Lustre: server umount lustre-MDT0001 complete [ 8970.094460] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 8975.652295] Key type lgssc unregistered [ 8976.062491] LNet: 218305:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8976.077459] LNetError: 218305:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8976.096384] LNet: Removed LNI 192.168.204.101@tcp [ 8977.336636] Key type .llcrypt unregistered [ 8977.340484] Key type ._llcrypt unregistered [ 9012.427681] Key type ._llcrypt registered [ 9012.433581] Key type .llcrypt registered [ 9012.646251] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 9013.798025] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9013.825075] alg: No test for adler32 (adler32-zlib) [ 9015.047436] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [ 9015.340858] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 9017.095451] Key type lgssc registered [ 9018.157605] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9027.073729] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9027.102141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9028.808811] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9034.478669] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 9039.330586] LustreError: 219575:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9039.356425] LustreError: 219575:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9044.458375] LustreError: 219576:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9045.940979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9046.562241] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9052.763444] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 9062.662571] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9071.278274] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9080.838773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9081.380978] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9090.223245] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 9098.832082] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 9100.447149] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9114.965892] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9115.272240] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9115.776918] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9125.806969] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 9140.467925] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9142.807546] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 9155.940759] Lustre: Permanently deactivating lustre-OST0001 [ 9155.984918] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 9161.891429] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 9173.343287] Lustre: Permanently reactivating lustre-OST0001 [ 9173.368120] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 9173.378752] Lustre: Skipped 2 previous similar messages [ 9182.789173] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9182.799075] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 9182.804512] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 9182.826171] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9182.855990] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 9188.322795] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9188.345027] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9188.351341] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9188.383995] Lustre: Skipped 2 previous similar messages [ 9188.414540] Lustre: Skipped 1 previous similar message [ 9189.981381] Lustre: server umount lustre-OST0001 complete [ 9193.441885] LustreError: 221400:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9193.494968] LustreError: 221400:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9196.029260] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9196.045233] LustreError: Skipped 1 previous similar message [ 9196.047249] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9196.079694] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9198.568226] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9198.569713] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9198.577641] LustreError: 221400:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9198.634910] Lustre: Skipped 1 previous similar message [ 9203.688102] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9203.694294] LustreError: 221400:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9203.708245] Lustre: Skipped 1 previous similar message [ 9203.738831] LustreError: 221400:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9208.801486] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9208.816294] Lustre: Skipped 1 previous similar message [ 9210.335132] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9210.881209] Lustre: server umount lustre-OST0000 complete [ 9219.043326] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9219.068068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9219.084025] Lustre: Skipped 1 previous similar message [ 9223.037026] Lustre: server umount lustre-MDT0000 complete [ 9224.163854] LustreError: 219575:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9224.206045] LustreError: 219575:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9228.233996] LustreError: 219557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786044157 with bad export cookie 5444137471502963919 [ 9228.243603] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9228.254447] LustreError: 219557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9229.138710] Lustre: server umount lustre-MDT0001 complete [ 9249.560274] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 9255.164366] Key type lgssc unregistered [ 9255.756169] LNet: 224657:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9255.764304] LNetError: 224657:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9255.798090] LNet: Removed LNI 192.168.204.101@tcp [ 9257.289674] Key type .llcrypt unregistered [ 9257.292071] Key type ._llcrypt unregistered [ 9317.751071] Key type ._llcrypt registered [ 9317.757148] Key type .llcrypt registered [ 9318.038356] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 9319.797034] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9319.814441] alg: No test for adler32 (adler32-zlib) [ 9321.107623] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [ 9321.294296] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 9322.959147] Key type lgssc registered [ 9324.835806] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9334.890683] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9334.922475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9336.380084] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9336.432380] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9336.705804] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9343.531684] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 9354.605081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9354.779520] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9354.810720] Lustre: 226866:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9355.224042] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9361.613243] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 9372.342162] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9380.787385] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9390.570056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9390.929170] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9391.230142] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9398.610920] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 9404.086156] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 9419.632179] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9426.435464] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 9428.640599] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 9438.083123] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 9444.700781] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 9457.597861] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 9457.609711] Lustre: Skipped 1 previous similar message [ 9478.393125] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 9478.403772] Lustre: Skipped 2 previous similar messages [ 9513.788174] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 9513.800572] Lustre: Skipped 3 previous similar messages [ 9585.479101] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 9585.483319] Lustre: Skipped 7 previous similar messages [ 9608.672704] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9608.690764] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9608.701893] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9608.713856] Lustre: Skipped 1 previous similar message [ 9608.734083] Lustre: Skipped 1 previous similar message [ 9613.480080] Lustre: server umount lustre-OST0000 complete [ 9621.472718] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9621.482331] LustreError: Skipped 1 previous similar message [ 9621.490412] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9621.514903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9624.033344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9624.034726] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9626.492027] Lustre: server umount lustre-MDT0000 complete [ 9632.244781] LustreError: 226859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786044561 with bad export cookie 8273873512585672705 [ 9632.250258] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9632.259849] LustreError: 226859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9633.130177] Lustre: server umount lustre-MDT0001 complete [ 9655.760607] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 9659.975679] Key type lgssc unregistered [ 9660.357346] LNet: 231680:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9660.366464] LNetError: 231680:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9660.403631] LNet: Removed LNI 192.168.204.101@tcp [ 9661.858157] Key type .llcrypt unregistered [ 9661.863062] Key type ._llcrypt unregistered [ 9696.611409] Key type ._llcrypt registered [ 9696.613841] Key type .llcrypt registered [ 9696.712491] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 9697.994823] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9698.363328] alg: No test for adler32 (adler32-zlib) [ 9699.602406] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [ 9700.018199] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 9701.783571] Key type lgssc registered [ 9703.518270] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9717.304849] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9717.377511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9719.386037] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9725.852090] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 9728.495035] LustreError: 233097:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9728.543512] LustreError: 233097:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9733.611564] LustreError: 233098:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9738.727076] LustreError: 233097:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9739.339360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9739.888918] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9746.597849] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 9756.896988] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9764.868024] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9775.189480] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9775.830871] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9785.014280] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 9789.116125] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 9795.951430] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9820.133876] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9820.146752] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9820.164161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9820.178270] Lustre: Skipped 1 previous similar message [ 9820.201560] Lustre: Skipped 1 previous similar message [ 9825.249186] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9825.270256] Lustre: Skipped 1 previous similar message [ 9830.386713] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9830.411029] Lustre: Skipped 1 previous similar message [ 9831.391547] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9831.690226] Lustre: server umount lustre-OST0000 complete [ 9840.608600] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9840.645167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9840.656730] Lustre: Skipped 1 previous similar message [ 9844.021457] Lustre: server umount lustre-MDT0000 complete [ 9845.732923] LustreError: 233097:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9845.748104] LustreError: 233097:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9848.124929] LustreError: 233079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786044777 with bad export cookie 13179208426125822635 [ 9848.128392] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9848.141398] LustreError: 233079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9848.879143] Lustre: server umount lustre-MDT0001 complete [ 9866.334523] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 9869.979315] Key type lgssc unregistered [ 9870.360620] LNet: 236800:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9870.379307] LNetError: 236800:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9870.400298] LNet: Removed LNI 192.168.204.101@tcp [ 9871.913579] Key type .llcrypt unregistered [ 9871.924390] Key type ._llcrypt unregistered [ 9904.256656] Key type ._llcrypt registered [ 9904.260305] Key type .llcrypt registered [ 9904.384519] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 9905.733131] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9906.057880] alg: No test for adler32 (adler32-zlib) [ 9907.474788] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [ 9907.830091] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 9909.615177] Key type lgssc registered [ 9911.277046] Lustre: Echo OBD driver; http://www.lustre.org/ [10187.840278] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [10191.468822] Key type lgssc unregistered [10191.761471] LNet: 238833:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10191.768715] LNetError: 238833:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10191.792588] LNet: Removed LNI 192.168.204.101@tcp [10193.173430] Key type .llcrypt unregistered [10193.175245] Key type ._llcrypt unregistered [10235.835773] Key type ._llcrypt registered [10235.840753] Key type .llcrypt registered [10236.009416] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_hostid [10257.802815] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [10259.067757] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10259.457141] alg: No test for adler32 (adler32-zlib) [10260.812508] Lustre: Lustre: Build Version: 2.17.56_52_g55149c9 [10261.106237] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [10262.815327] Key type lgssc registered [10264.297354] Lustre: Echo OBD driver; http://www.lustre.org/ [10306.270571] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 9866 sec ========== 15:40:33 (1786045233) [10308.955619] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 15:40:35 (1786045235) === [10313.610635] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 15:40:40 (1786045240) === [10374.241424] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [10380.425550] Key type lgssc unregistered [10381.047632] LNet: 244082:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10381.055206] LNetError: 244082:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10381.092470] LNet: Removed LNI 192.168.204.101@tcp [10383.003803] Key type .llcrypt unregistered [10383.009943] Key type ._llcrypt unregistered