[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 617520751 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002232] x2apic enabled [ 0.004004] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.008310] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009015] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010010] pid_max: default: 32768 minimum: 301 [ 0.011127] LSM: Security Framework initializing [ 0.012049] Yama: becoming mindful. [ 0.013033] SELinux: Initializing. [ 0.014067] *** VALIDATE selinux *** [ 0.022241] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026676] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028027] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029113] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030101] *** VALIDATE tmpfs *** [ 0.032358] *** VALIDATE proc *** [ 0.033216] *** VALIDATE cgroup *** [ 0.034006] *** VALIDATE cgroup2 *** [ 0.035248] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036145] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038028] Spectre V2 : User space: Vulnerable [ 0.039007] Speculative Store Bypass: Vulnerable [ 0.043212] debug: unmapping init [mem 0xffffffff85a59000-0xffffffff85a60fff] [ 0.045908] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046768] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047021] ... version: 2 [ 0.048009] ... bit width: 48 [ 0.049007] ... generic registers: 4 [ 0.050011] ... value mask: 0000ffffffffffff [ 0.051015] ... max period: 00007fffffffffff [ 0.052020] ... fixed-purpose events: 3 [ 0.053013] ... event mask: 000000070000000f [ 0.054292] rcu: Hierarchical SRCU implementation. [ 0.056571] smp: Bringing up secondary CPUs ... [ 0.057678] x86: Booting SMP configuration: [ 0.058036] .... node #0, CPUs: #1 #2 #3 [ 0.062091] smp: Brought up 1 node, 4 CPUs [ 0.064016] smpboot: Max logical packages: 1 [ 0.065014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.140274] node 0 deferred pages initialised in 73ms [ 0.143114] devtmpfs: initialized [ 0.144231] x86/mm: Memory block size: 128MB [ 0.146674] gcov: version magic: 0x41383552 [ 0.148324] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.151074] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.153203] pinctrl core: initialized pinctrl subsystem [ 0.154157] [ 0.154684] ************************************************************* [ 0.155012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.156011] ** ** [ 0.157010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.158010] ** ** [ 0.159013] ** This means that this kernel is built to expose internal ** [ 0.160010] ** IOMMU data structures, which may compromise security on ** [ 0.161015] ** your system. ** [ 0.162013] ** ** [ 0.163010] ** If you see this message and you are not debugging the ** [ 0.164010] ** kernel, report this immediately to your vendor! ** [ 0.165011] ** ** [ 0.166010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.167010] ************************************************************* [ 0.168678] NET: Registered protocol family 16 [ 0.169360] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.170050] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.171054] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.173091] cpuidle: using governor menu [ 0.174898] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.176569] PCI: Using configuration type 1 for base access [ 0.179139] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.188074] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.189033] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.191065] cryptd: max_cpu_qlen set to 1000 [ 0.194259] ACPI: Added _OSI(Module Device) [ 0.195010] ACPI: Added _OSI(Processor Device) [ 0.196014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.197009] ACPI: Added _OSI(Processor Aggregator Device) [ 0.201021] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.205229] ACPI: Interpreter enabled [ 0.206052] ACPI: PM: (supports S0 S3 S4 S5) [ 0.207020] ACPI: Using IOAPIC for interrupt routing [ 0.208084] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.209385] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.218490] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.219027] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.220013] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.221068] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.223303] acpiphp: Slot [2] registered [ 0.224089] acpiphp: Slot [5] registered [ 0.225128] acpiphp: Slot [6] registered [ 0.226099] acpiphp: Slot [7] registered [ 0.227075] acpiphp: Slot [8] registered [ 0.228063] acpiphp: Slot [9] registered [ 0.229000] acpiphp: Slot [10] registered [ 0.230079] acpiphp: Slot [3] registered [ 0.231082] acpiphp: Slot [4] registered [ 0.232063] acpiphp: Slot [11] registered [ 0.233062] acpiphp: Slot [12] registered [ 0.234061] acpiphp: Slot [13] registered [ 0.235061] acpiphp: Slot [14] registered [ 0.236070] acpiphp: Slot [15] registered [ 0.237078] acpiphp: Slot [16] registered [ 0.238056] acpiphp: Slot [17] registered [ 0.239057] acpiphp: Slot [18] registered [ 0.240056] acpiphp: Slot [19] registered [ 0.241053] acpiphp: Slot [20] registered [ 0.242074] acpiphp: Slot [21] registered [ 0.243059] acpiphp: Slot [22] registered [ 0.244065] acpiphp: Slot [23] registered [ 0.245102] acpiphp: Slot [24] registered [ 0.246060] acpiphp: Slot [25] registered [ 0.246960] acpiphp: Slot [26] registered [ 0.248056] acpiphp: Slot [27] registered [ 0.249005] acpiphp: Slot [28] registered [ 0.250084] acpiphp: Slot [29] registered [ 0.251061] acpiphp: Slot [30] registered [ 0.253061] acpiphp: Slot [31] registered [ 0.254066] PCI host bridge to bus 0000:00 [ 0.255011] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.256011] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.258010] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.260013] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.261011] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.263014] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.264148] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.266779] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.269008] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.277013] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.282098] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.285015] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.288012] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.291019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.295164] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.296642] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.299026] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.301724] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.306012] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.317012] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.321013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.326020] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.339014] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.353015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.386016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.403293] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.417015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.437019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.486018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.508466] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.528018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.545015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.597015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.622085] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.640014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.658017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.712020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.735797] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.753015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.767016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.810093] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.825358] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.833014] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.849114] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.917021] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.941668] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.949684] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.954698] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.959323] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.964937] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.976102] iommu: Default domain type: Passthrough [ 0.979128] SCSI subsystem initialized [ 0.983028] ACPI: bus type USB registered [ 0.986093] usbcore: registered new interface driver usbfs [ 0.990107] usbcore: registered new interface driver hub [ 0.993341] usbcore: registered new device driver usb [ 0.997090] pps_core: LinuxPPS API ver. 1 registered [ 1.002010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.010243] PTP clock support registered [ 1.016138] EDAC MC: Ver: 3.0.0 [ 1.017068] PCI: Using ACPI for IRQ routing [ 1.017873] NetLabel: Initializing [ 1.018000] NetLabel: domain hash size = 128 [ 1.021010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.024070] NetLabel: unlabeled traffic allowed by default [ 1.029044] vgaarb: loaded [ 1.037362] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.040009] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.056977] clocksource: Switched to clocksource kvm-clock [ 1.329213] VFS: Disk quotas dquot_6.6.0 [ 1.330732] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.340600] *** VALIDATE ramfs *** [ 1.344751] *** VALIDATE hugetlbfs *** [ 1.349827] pnp: PnP ACPI init [ 1.356143] pnp: PnP ACPI: found 6 devices [ 1.460768] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.464833] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.473820] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.480192] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.488072] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.492578] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.508777] NET: Registered protocol family 2 [ 1.515910] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.533641] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.539821] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.555768] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.576532] TCP: Hash tables configured (established 65536 bind 65536) [ 1.581644] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.590757] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.597159] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.606837] NET: Registered protocol family 1 [ 1.616546] RPC: Registered named UNIX socket transport module. [ 1.621926] RPC: Registered udp transport module. [ 1.627621] RPC: Registered tcp transport module. [ 1.629811] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.638064] NET: Registered protocol family 44 [ 1.644910] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.661143] pci 0000:00:00.0: quirk_natoma+0x0/0x20 took 15852 usecs [ 1.673119] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.688072] pci 0000:00:00.0: quirk_passive_release+0x0/0x90 took 14630 usecs [ 1.697611] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.715076] pci 0000:00:01.0: quirk_isa_dma_hangs+0x0/0x20 took 17055 usecs [ 1.722759] PCI: CLS 0 bytes, default 64 [ 1.730582] Unpacking initramfs... [ 7.452455] debug: unmapping init [mem 0xffff94ce7cc54000-0xffff94ce7ffbffff] [ 7.472150] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 7.484293] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 7.500967] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 9.310617] Initialise system trusted keyrings [ 9.313132] Key type blacklist registered [ 9.316347] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 9.329642] zbud: loaded [ 9.334270] *** VALIDATE nfs *** [ 9.335523] *** VALIDATE nfs4 *** [ 9.338498] pstore: using deflate compression [ 9.361477] Platform Keyring initialized [ 9.825450] NET: Registered protocol family 38 [ 9.827314] Key type asymmetric registered [ 9.828611] Asymmetric key parser 'x509' registered [ 9.830486] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 9.833780] io scheduler mq-deadline registered [ 9.835484] io scheduler kyber registered [ 9.840932] io scheduler bfq registered [ 9.843337] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 9.847204] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 9.850456] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 9.855358] ACPI: Power Button [PWRF] [ 9.860929] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 9.871510] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 9.895962] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 9.922471] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 9.967536] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 10.015289] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 10.070744] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 10.083344] Non-volatile memory driver v1.3 [ 10.086307] Linux agpgart interface v0.103 [ 10.158487] virtio_blk virtio1: [vda] 145920 512-byte logical blocks (74.7 MB/71.3 MiB) [ 10.160953] vda: detected capacity change from 0 to 74711040 [ 10.181087] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 10.184827] vdb: detected capacity change from 0 to 1073741824 [ 10.199787] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 10.202860] vdc: detected capacity change from 0 to 2621440000 [ 10.237537] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 10.244328] vdd: detected capacity change from 0 to 2621440000 [ 10.278271] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 10.286204] vde: detected capacity change from 0 to 4294967296 [ 10.327950] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 10.336182] vdf: detected capacity change from 0 to 4294967296 [ 10.355557] libphy: Fixed MDIO Bus: probed [ 10.365046] usbcore: registered new interface driver usbserial_generic [ 10.367833] usbserial: USB Serial support registered for generic [ 10.370508] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 10.375708] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 10.377835] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 10.380727] mousedev: PS/2 mouse device common for all mice [ 10.384886] rtc_cmos 00:05: RTC can wake from S4 [ 10.388465] rtc_cmos 00:05: registered as rtc0 [ 10.390342] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 10.394469] intel_pstate: CPU model not supported [ 10.398378] hid: raw HID events driver (C) Jiri Kosina [ 10.400830] usbcore: registered new interface driver usbhid [ 10.403565] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 10.406952] usbhid: USB HID core driver [ 10.415952] drop_monitor: Initializing network drop monitor service [ 10.418337] Initializing XFRM netlink socket [ 10.419994] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 10.425569] NET: Registered protocol family 10 [ 10.434632] Segment Routing with IPv6 [ 10.436571] NET: Registered protocol family 17 [ 10.439111] mpls_gso: MPLS GSO support [ 10.447086] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 10.451338] RAS: Correctable Errors collector initialized. [ 10.455767] AVX version of gcm_enc/dec engaged. [ 10.460099] AES CTR mode by8 optimization enabled [ 10.744325] sched_clock: Marking stable (10744288961, 0)->(11677622993, -933334032) [ 10.759848] registered taskstats version 1 [ 10.769479] Loading compiled-in X.509 certificates [ 10.771606] zswap: loaded using pool lzo/zbud [ 10.847195] Key type big_key registered [ 10.899477] Key type encrypted registered [ 10.905740] ima: No TPM chip found, activating TPM-bypass! [ 10.921561] ima: Allocated hash algorithm: sha1 [ 10.930389] ima: No architecture policies found [ 10.936139] evm: Initialising EVM extended attributes: [ 10.944707] evm: security.selinux [ 10.949974] evm: security.ima [ 10.956638] evm: security.capability [ 10.968215] evm: HMAC attrs: 0x1 [ 10.981819] rtc_cmos 00:05: setting system clock to 2026-07-30 07:28:28 UTC (1785396508) [ 11.005558] debug: unmapping init [mem 0xffffffff86a03000-0xffffffff86bfffff] [ 11.030689] debug: unmapping init [mem 0xffffffff85782000-0xffffffff85a58fff] [ 11.044103] Write protecting the kernel read-only data: 28672k [ 11.059913] debug: unmapping init [mem 0xffffffff83e03000-0xffffffff83ffffff] [ 11.072978] debug: unmapping init [mem 0xffffffff84714000-0xffffffff847fffff] [ 11.254898] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.297632] systemd[1]: Detected virtualization kvm. [ 11.304052] systemd[1]: Detected architecture x86-64. [ 11.311943] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 11.415605] systemd[1]: No hostname configured. [ 11.420814] systemd[1]: Set hostname to . [ 11.427744] random: systemd: uninitialized urandom read (16 bytes read) [ 11.435954] systemd[1]: Initializing machine ID from random generator. [ 11.802978] random: systemd: uninitialized urandom read (16 bytes read) [ 11.806813] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 11.814454] random: systemd: uninitialized urandom read (16 bytes read) [ 11.825922] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 11.837419] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. [ OK ] Reached target Sockets. Starting Journal Service... Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 14.043425] device-mapper: uevent: version 1.0.3 [ 14.046907] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 16.625569] virtio_net virtio0 ens2: renamed from eth0 [ 17.667927] scsi host0: ata_piix [ 18.073212] scsi host1: ata_piix [ 18.088610] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 18.110493] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 22.532124] random: crng init done [ 22.538284] random: 7 urandom warning(s) missed due to ratelimiting [ 25.821320] dracut-initqueue[580]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 30.231402] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ 30.634770] hrtimer: interrupt took 39719367 ns [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 35.682959] printk: systemd: 26 output lines suppressed due to ratelimiting [ 36.608090] SELinux: Disabled at runtime. [ 36.707525] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 36.720041] systemd[1]: Detected virtualization kvm. [ 36.721852] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 38.507142] systemd[1]: initrd-switch-root.service: Succeeded. [ 38.521934] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 38.554184] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 38.565984] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 38.580065] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 38.647169] systemd[1]: Starting Journal Service... Starting Journal Service... [ 38.671059] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. Starting Create list of required st…ce nodes for the current kernel... Mounting POSIX Message Queue File System... [ OK ] Listening on udev Control Socket. Mounting Huge Pages File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-getty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Apply Kernel Variables... Activating swap /dev/disk/by-label/SWAP... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Initrd Root File System. Mounting Kernel Debug File System... [ 39.398059] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target Paths. [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... Starting Create Static Device Nodes in /dev... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 41.377839] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 43.111881] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 43.369321] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 45.950058] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 46.555197] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (9s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit) [*** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (11s / no limit) [ ***] A start job is running for Configur…only root support (11s / no limit) [ **] A start job is running for Configur…only root support (12s / no limit) [ *] A start job is running for Configur…only root support (12s / no limit) [ **] A start job is running for Configur…only root support (13s / no limit) [ ***] A start job is running for Configur…only root support (13s / no limit) [ *** ] A start job is running for Configur…only root support (14s / no limit) [ *** ] A start job is running for Configur…only root support (14s / no limit) [*** ] A start job is running for Configur…only root support (15s / no limit) [** ] A start job is running for Configur…only root support (16s / no limit)[ 54.420815] Key type dns_resolver registered [* ] A start job is running for Configur…only root support (16s / no limit) [** ] A start job is running for Configur…only root support (17s / no limit) [*** ] A start job is running for Configur…only root support (17s / no limit) [ *** ] A start job is running for Configur…only root support (18s / no limit) [ *** ] A start job is running for Configur…only root support (18s / no limit) [ ***] A start job is running for Configur…only root support (19s / no limit) [ **] A start job is running for Configur…only root support (19s / no limit)[ 58.223145] NFS: Registering the id_resolver key type [ 58.250711] Key type id_resolver registered [ 58.251814] Key type id_legacy registered [ *] A start job is running for Configur…only root support (20s / no limit) [ **] A start job is running for Configur…only root support (20s / no limit) [ ***] A start job is running for Configur…only root support (21s / no limit) [ *** ] A start job is running for Configur…only root support (21s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg213-server login: [ 131.521758] libcfs: loading out-of-tree module taints kernel. [ 131.633317] Key type ._llcrypt registered [ 131.640969] Key type .llcrypt registered [ 131.853648] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_hostid [ 157.225027] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 159.134306] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 159.146700] alg: No test for adler32 (adler32-zlib) [ 161.083754] Lustre: Lustre: Build Version: 2.17.56_2_gaec046b [ 162.450532] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 164.489257] Key type lgssc registered [ 167.605159] Lustre: Echo OBD driver; http://www.lustre.org/ [ 191.667135] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 247.897886] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 264.237862] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 264.268992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 265.673321] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 265.723123] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 265.806912] Lustre: lustre-MDT0000: new disk, initializing [ 265.895430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 265.914971] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 271.891789] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 290.535532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 290.735154] Lustre: 6519:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 290.864687] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 290.881448] Lustre: Skipped 1 previous similar message [ 291.116760] Lustre: lustre-MDT0001: new disk, initializing [ 291.236032] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 291.268914] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 291.284342] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 298.811813] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 306.796950] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 324.758153] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 325.226874] Lustre: lustre-OST0000: new disk, initializing [ 325.234867] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 325.247895] Lustre: 8457:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 325.328293] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 331.865159] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 331.885736] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 332.011703] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 334.414546] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 354.391978] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 354.793544] Lustre: lustre-OST0001: new disk, initializing [ 354.801292] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 354.813505] Lustre: 9532:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 354.961867] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 362.051105] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 362.064239] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 362.200159] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 365.310352] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 387.924379] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 399.022781] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 410.397198] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing check_logdir /tmp/testlogs/ [ 419.282580] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing yml_node [ 426.827493] Lustre: DEBUG MARKER: Client: 2.17.56.2 [ 430.981255] Lustre: DEBUG MARKER: MDS: 2.17.56.2 [ 434.449719] Lustre: DEBUG MARKER: OSS: 2.17.56.2 [ 436.767770] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Jul 30 03:35:31 EDT 2026 [ 460.081838] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 461.873755] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 485.344492] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 485.345933] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 485.369018] Lustre: Skipped 1 previous similar message [ 485.386240] Lustre: Skipped 2 previous similar messages [ 491.045195] Lustre: server umount lustre-MDT0000 complete [ 495.404684] LustreError: 8456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785396992 with bad export cookie 11273875244290691663 [ 495.408463] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 495.420341] LustreError: 8456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 495.583720] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 495.588413] LustreError: 6523:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 495.588544] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 495.588548] Lustre: Skipped 1 previous similar message [ 495.609121] Lustre: Skipped 3 previous similar messages [ 495.654106] LustreError: 6523:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 500.704817] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 500.705582] LustreError: 10317:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 500.716052] Lustre: Skipped 2 previous similar messages [ 500.750340] LustreError: 10317:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 502.163753] Lustre: server umount lustre-MDT0001 complete [ 508.384054] Lustre: server umount lustre-OST0000 complete [ 514.555088] Lustre: server umount lustre-OST0001 complete [ 527.193413] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_hostid [ 545.081219] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 600.391613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 600.763436] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 600.785320] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 600.863451] Lustre: lustre-MDT0000: new disk, initializing [ 600.961725] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 601.003468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 608.121337] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 622.078242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 622.334681] Lustre: 15115:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 622.516460] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 622.524307] Lustre: Skipped 1 previous similar message [ 622.645184] Lustre: lustre-MDT0001: new disk, initializing [ 622.786282] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 622.855189] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 622.877976] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 629.505835] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 642.976465] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 650.863975] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 660.097902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 660.539589] Lustre: lustre-OST0000: new disk, initializing [ 660.548458] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 660.557892] Lustre: 17090:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 660.771305] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 662.588433] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 662.606676] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 662.821177] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 669.718186] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 681.236539] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 687.963973] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 688.255880] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 694.355155] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 694.745344] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 698.853130] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 698.856009] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 698.868040] LustreError: Skipped 1 previous similar message [ 698.868733] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 698.900550] Lustre: Skipped 1 previous similar message [ 703.971709] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 703.978290] Lustre: Skipped 2 previous similar messages [ 711.135795] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 711.680583] Lustre: server umount lustre-OST0000 complete [ 719.329151] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 719.337318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 719.344351] Lustre: Skipped 1 previous similar message [ 719.363835] Lustre: Skipped 3 previous similar messages [ 724.345785] Lustre: server umount lustre-MDT0000 complete [ 724.454127] LustreError: 15129:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 724.470902] LustreError: 15129:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 729.577984] LustreError: 15978:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 729.617930] LustreError: 15978:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 729.802188] LustreError: 15108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785397227 with bad export cookie 11273875244290693196 [ 729.820077] LustreError: 15108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 729.821715] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 730.658376] Lustre: server umount lustre-MDT0001 complete [ 736.704280] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 03:40:31 (1785397231) [ 748.298182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 749.199263] LustreError: 19361:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 749.418762] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 757.279398] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 758.767424] LustreError: 19362:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 758.802634] LustreError: 19362:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 770.869991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 778.395728] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 788.787117] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 797.129549] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 806.654363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 813.829874] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 824.994159] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 838.114901] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 838.138432] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 838.165465] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 838.744367] Lustre: server umount lustre-OST0000 complete [ 853.730626] Lustre: server umount lustre-MDT0000 complete [ 854.502205] LustreError: 20377:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 854.526991] LustreError: 20377:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 858.654123] LustreError: 19341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785397356 with bad export cookie 11273875244290694071 [ 858.674369] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 859.299760] Lustre: server umount lustre-MDT0001 complete [ 873.987531] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 879.109807] Key type lgssc unregistered [ 879.683723] LNet: 22899:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 879.709427] LNetError: 22899:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 879.740795] LNet: Removed LNI 192.168.202.113@tcp [ 882.122201] Key type .llcrypt unregistered [ 882.128844] Key type ._llcrypt unregistered [ 913.613334] Key type ._llcrypt registered [ 913.615779] Key type .llcrypt registered [ 913.836869] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 915.797792] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 915.945916] alg: No test for adler32 (adler32-zlib) [ 917.340640] Lustre: Lustre: Build Version: 2.17.56_2_gaec046b [ 917.727632] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 919.471146] Key type lgssc registered [ 921.343783] Lustre: Echo OBD driver; http://www.lustre.org/ [ 933.949140] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 933.993826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 935.829561] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 942.655694] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 946.147826] LustreError: 24314:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 946.162661] LustreError: 24314:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 951.271456] LustreError: 24315:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 953.141025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 953.755740] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 961.044508] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 971.695964] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 979.011176] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 987.678467] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 988.360329] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 996.454814] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 1005.879933] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1006.952644] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1030.626542] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1030.654400] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1030.692701] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1031.652243] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1031.667238] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1031.668018] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1031.671752] Lustre: Skipped 1 previous similar message [ 1034.155186] Lustre: server umount lustre-OST0000 complete [ 1040.864591] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1040.875630] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1040.903910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1044.727536] Lustre: server umount lustre-MDT0000 complete [ 1047.015804] LustreError: 26760:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1047.042405] LustreError: 26760:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1050.007452] LustreError: 24294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785397547 with bad export cookie 5473233746668673505 [ 1050.040472] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1050.577976] Lustre: server umount lustre-MDT0001 complete [ 1070.646143] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 1080.492800] Key type lgssc unregistered [ 1081.621895] LNet: 28317:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1081.659064] LNetError: 28317:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1081.765091] LNet: Removed LNI 192.168.202.113@tcp [ 1083.900782] Key type .llcrypt unregistered [ 1083.902323] Key type ._llcrypt unregistered [ 1121.350625] Key type ._llcrypt registered [ 1121.355366] Key type .llcrypt registered [ 1121.472416] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 1122.855099] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1122.922074] alg: No test for adler32 (adler32-zlib) [ 1124.289396] Lustre: Lustre: Build Version: 2.17.56_2_gaec046b [ 1124.728770] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 1126.519177] Key type lgssc registered [ 1127.953488] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1139.397947] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1139.449289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1141.205237] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1147.807304] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 1151.463467] LustreError: 29734:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1151.491403] LustreError: 29734:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1156.588544] LustreError: 29735:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1160.378688] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1161.079498] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1167.278185] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 1178.415869] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1189.007197] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1213.421232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1214.228672] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1223.246156] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 1233.970127] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1235.769476] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1251.304183] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1251.306040] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1251.314908] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1251.320246] LustreError: Skipped 1 previous similar message [ 1251.366256] Lustre: Skipped 1 previous similar message [ 1254.793056] Lustre: server umount lustre-OST0000 complete [ 1261.558814] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1261.576320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1261.594342] Lustre: Skipped 1 previous similar message [ 1261.619281] Lustre: Skipped 2 previous similar messages [ 1266.680492] LustreError: 29734:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1267.075345] Lustre: server umount lustre-MDT0000 complete [ 1270.771189] LustreError: 30463:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1270.802850] LustreError: 30463:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1272.338395] LustreError: 29715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785397769 with bad export cookie 15084298776908793295 [ 1272.344671] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1272.357640] LustreError: 29715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1272.974898] Lustre: server umount lustre-MDT0001 complete [ 1290.095311] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 1294.617754] Key type lgssc unregistered [ 1295.090117] LNet: 33641:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1295.096848] LNetError: 33641:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1295.122850] LNet: Removed LNI 192.168.202.113@tcp [ 1296.821688] Key type .llcrypt unregistered [ 1296.823257] Key type ._llcrypt unregistered [ 1330.365193] Key type ._llcrypt registered [ 1330.366751] Key type .llcrypt registered [ 1330.436451] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 1331.440979] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1331.480690] alg: No test for adler32 (adler32-zlib) [ 1332.502259] Lustre: Lustre: Build Version: 2.17.56_2_gaec046b [ 1332.824363] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 1334.624898] Key type lgssc registered [ 1336.614457] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1348.746480] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1348.788905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1350.660263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1358.709584] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 1360.892857] LustreError: 35060:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1360.921045] LustreError: 35060:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1365.987460] LustreError: 35059:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1370.007590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1370.633096] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1376.336720] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 1389.376291] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1399.448903] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1408.927947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1409.414403] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1420.715562] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 1420.731694] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1429.928752] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1445.344609] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1445.369526] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1445.413326] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1446.370275] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1446.375918] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1446.426380] Lustre: Skipped 1 previous similar message [ 1448.595385] Lustre: server umount lustre-OST0000 complete [ 1456.609153] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1456.610703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1456.640992] Lustre: Skipped 1 previous similar message [ 1456.657331] Lustre: Skipped 1 previous similar message [ 1460.632400] Lustre: server umount lustre-MDT0000 complete [ 1461.728637] LustreError: 35055:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1461.762912] LustreError: 35055:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1465.885271] LustreError: 35040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785397963 with bad export cookie 1213870746415264436 [ 1465.895967] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1466.589587] Lustre: server umount lustre-MDT0001 complete [ 1487.860789] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 1493.668810] Key type lgssc unregistered [ 1494.129661] LNet: 38620:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1494.136913] LNetError: 38620:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1494.163931] LNet: Removed LNI 192.168.202.113@tcp [ 1496.216338] Key type .llcrypt unregistered [ 1496.222934] Key type ._llcrypt unregistered [ 1527.969635] Key type ._llcrypt registered [ 1527.972250] Key type .llcrypt registered [ 1528.154409] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 1529.484380] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1529.552222] alg: No test for adler32 (adler32-zlib) [ 1530.750203] Lustre: Lustre: Build Version: 2.17.56_2_gaec046b [ 1531.074602] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 1532.744308] Key type lgssc registered [ 1534.622468] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1544.560882] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1544.590203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1546.263669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1552.612513] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 1556.462536] LustreError: 40036:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1556.490496] LustreError: 40036:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1561.577787] LustreError: 40037:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1563.852053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1564.386661] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1571.692090] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 1582.184472] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1590.440856] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1600.191966] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1600.908712] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1609.794542] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 1616.261394] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1620.267753] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1629.666038] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1629.690140] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1629.720745] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1631.721280] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1631.726264] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1631.756863] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1635.258346] Lustre: server umount lustre-OST0000 complete [ 1658.968565] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1658.974930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1658.989950] Lustre: Skipped 1 previous similar message [ 1659.020744] Lustre: Skipped 3 previous similar messages [ 1663.010222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1663.019142] Lustre: Skipped 1 previous similar message [ 1665.690698] Lustre: server umount lustre-MDT0000 complete [ 1668.076056] LustreError: 40037:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1668.092649] LustreError: 40037:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1670.741156] LustreError: 40016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785398168 with bad export cookie 4953267377282097420 [ 1670.743421] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1670.754381] LustreError: 40016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1677.297758] Lustre: server umount lustre-MDT0001 complete [ 1691.800448] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 1695.910598] Key type lgssc unregistered [ 1696.291030] LNet: 43749:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1696.304791] LNetError: 43749:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1697.338952] LNet: Removed LNI 192.168.202.113@tcp [ 1698.875078] Key type .llcrypt unregistered [ 1698.881099] Key type ._llcrypt unregistered [ 1731.646437] Key type ._llcrypt registered [ 1731.669249] Key type .llcrypt registered [ 1731.840546] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 1734.045092] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1734.195334] alg: No test for adler32 (adler32-zlib) [ 1735.411782] Lustre: Lustre: Build Version: 2.17.56_2_gaec046b [ 1735.894725] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 1737.615270] Key type lgssc registered [ 1739.699798] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1751.617560] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1751.696775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1753.646098] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1760.002704] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 1763.824877] LustreError: 45170:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1763.877121] LustreError: 45170:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1768.941905] LustreError: 45169:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1773.332269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1774.153487] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1781.100442] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 1791.064147] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1799.043125] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1808.600748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1809.176738] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1816.867772] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 1824.566177] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1828.165619] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1840.100404] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1840.113590] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1840.134050] Lustre: Skipped 1 previous similar message [ 1840.161265] Lustre: Skipped 2 previous similar messages [ 1841.583368] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.13@tcp (stopping) [ 1843.373022] Lustre: server umount lustre-MDT0000 complete [ 1845.224798] LustreError: 45170:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1845.249526] LustreError: 45170:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1848.553288] LustreError: 45150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785398346 with bad export cookie 2064776440014396790 [ 1848.586080] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1850.339740] LustreError: 45164:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1850.347689] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1850.354264] LustreError: 45164:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1850.371313] Lustre: Skipped 1 previous similar message [ 1850.375625] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1853.344808] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1853.353842] Lustre: Skipped 1 previous similar message [ 1855.585801] Lustre: server umount lustre-MDT0001 complete [ 1873.953119] Lustre: server umount lustre-OST0000 complete [ 1896.599668] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 1901.488490] Key type lgssc unregistered [ 1901.868116] LNet: 49035:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1901.873446] LNetError: 49035:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1901.900766] LNet: Removed LNI 192.168.202.113@tcp [ 1903.463927] Key type .llcrypt unregistered [ 1903.468491] Key type ._llcrypt unregistered [ 1939.884318] Key type ._llcrypt registered [ 1939.888434] Key type .llcrypt registered [ 1940.043503] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 1941.863781] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1941.983040] alg: No test for adler32 (adler32-zlib) [ 1943.083342] Lustre: Lustre: Build Version: 2.17.56_2_gaec046b [ 1943.443700] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 1945.139499] Key type lgssc registered [ 1947.332919] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1959.209767] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1959.255630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1974.879631] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1979.999444] LustreError: 50423:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.113@tcp: failed processing log, type 4: rc = -110 [ 2006.751926] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2017.020350] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2027.152653] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2194.889540] Lustre: server umount lustre-OST0000 complete [ 2218.995076] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 2222.119164] Key type lgssc unregistered [ 2222.516530] LNet: 52117:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2222.524494] LNetError: 52117:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2222.537627] LNet: Removed LNI 192.168.202.113@tcp [ 2223.563483] Key type .llcrypt unregistered [ 2223.568677] Key type ._llcrypt unregistered [ 2256.441957] Key type ._llcrypt registered [ 2256.451276] Key type .llcrypt registered [ 2256.632222] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 2257.444651] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2257.621085] alg: No test for adler32 (adler32-zlib) [ 2258.823634] Lustre: Lustre: Build Version: 2.17.56_2_gaec046b [ 2259.300643] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 2261.103159] Key type lgssc registered [ 2263.106218] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2275.271497] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2275.351030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2277.649676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2285.239296] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2288.109229] LustreError: 53534:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2288.140203] LustreError: 53534:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2293.221757] LustreError: 53535:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2296.111177] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2296.830481] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2302.067844] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2311.177304] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2318.688309] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2327.081781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2327.461060] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2336.337771] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2346.000787] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2349.046545] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 2352.098349] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2352.110841] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2352.138515] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2354.144806] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2354.161718] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2354.163775] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2355.650045] Lustre: server umount lustre-OST0000 complete [ 2363.361937] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2363.381690] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2363.411139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2363.420322] Lustre: Skipped 1 previous similar message [ 2367.887336] Lustre: server umount lustre-MDT0000 complete [ 2369.522414] LustreError: 54262:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2372.884050] LustreError: 53513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785398870 with bad export cookie 14825925007155233957 [ 2372.894620] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2372.899186] LustreError: 53513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2373.505131] Lustre: server umount lustre-MDT0001 complete [ 2393.791887] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 2398.358121] Key type lgssc unregistered [ 2398.920647] LNet: 57090:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2398.928770] LNetError: 57090:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2398.954921] LNet: Removed LNI 192.168.202.113@tcp [ 2400.085722] Key type .llcrypt unregistered [ 2400.089593] Key type ._llcrypt unregistered [ 2433.133735] Key type ._llcrypt registered [ 2433.139734] Key type .llcrypt registered [ 2433.320232] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 2435.445746] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2435.511882] alg: No test for adler32 (adler32-zlib) [ 2436.757307] Lustre: Lustre: Build Version: 2.17.56_2_gaec046b [ 2437.073802] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 2438.815386] Key type lgssc registered [ 2440.528233] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2451.055962] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2451.092184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2466.719356] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2471.839315] LustreError: 58478:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.113@tcp: failed processing log, type 4: rc = -110 [ 2498.527374] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2507.553590] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2517.349196] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2527.826294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2528.539287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2528.763978] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2537.981744] LustreError: 59517:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2538.050963] LustreError: 59517:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 2540.865535] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2543.076659] LustreError: 58503:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2543.102939] LustreError: 58503:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2548.201656] LustreError: 58504:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2548.228762] LustreError: 58504:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2553.317798] LustreError: 58503:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2553.340815] LustreError: 58503:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2556.012309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2556.816525] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2565.250815] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2577.096327] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2583.788672] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2587.622175] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2587.624875] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2587.637677] Lustre: Skipped 1 previous similar message [ 2587.645910] Lustre: Skipped 1 previous similar message [ 2592.068398] Lustre: server umount lustre-OST0000 complete [ 2623.460658] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2623.468732] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2623.472115] Lustre: Skipped 1 previous similar message [ 2623.489942] Lustre: Skipped 1 previous similar message [ 2623.503384] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2627.555070] LustreError: 59517:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2627.576109] LustreError: 59517:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2628.079304] Lustre: server umount lustre-MDT0000 complete [ 2634.600911] LustreError: 58484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785399132 with bad export cookie 10001337665345555274 [ 2634.641390] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2642.060577] Lustre: server umount lustre-MDT0001 complete [ 2660.649664] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 2664.765593] Key type lgssc unregistered [ 2665.363662] LNet: 62233:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2665.379790] LNetError: 62233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2665.428452] LNet: Removed LNI 192.168.202.113@tcp [ 2666.988640] Key type .llcrypt unregistered [ 2666.994306] Key type ._llcrypt unregistered [ 2711.619620] Key type ._llcrypt registered [ 2711.621597] Key type .llcrypt registered [ 2711.703581] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 2712.707053] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2712.771225] alg: No test for adler32 (adler32-zlib) [ 2714.441448] Lustre: Lustre: Build Version: 2.17.56_2_gaec046b [ 2715.052912] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 2716.911189] Key type lgssc registered [ 2719.054810] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2729.877273] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2729.910840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2731.627508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2737.338747] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2742.252437] LustreError: 63651:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2742.285813] LustreError: 63651:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2747.360374] LustreError: 63652:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2750.606253] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2751.468950] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2759.179637] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2770.605225] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2779.065521] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2787.945571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2788.681099] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2797.370331] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2801.971537] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2808.199615] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2836.961608] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2836.993442] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2837.013026] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2837.987789] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2837.988353] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2838.000656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2843.123065] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2843.149356] Lustre: Skipped 2 previous similar messages [ 2848.228892] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2848.238092] Lustre: Skipped 1 previous similar message [ 2850.271138] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2850.575693] Lustre: server umount lustre-OST0000 complete [ 2858.484052] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2858.497617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2858.510641] Lustre: Skipped 1 previous similar message [ 2858.537549] Lustre: Skipped 1 previous similar message [ 2862.719468] Lustre: server umount lustre-MDT0000 complete [ 2863.586872] LustreError: 64382:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2863.606295] LustreError: 64382:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2868.713452] LustreError: 63651:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2868.760357] LustreError: 63651:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2868.830823] LustreError: 65497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785399366 with bad export cookie 7915756978281430075 [ 2868.846480] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2868.853143] LustreError: 65497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2870.146513] Lustre: server umount lustre-MDT0001 complete [ 2901.507758] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 2906.971624] Key type lgssc unregistered [ 2907.549450] LNet: 67242:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2907.563386] LNetError: 67242:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2907.605174] LNet: Removed LNI 192.168.202.113@tcp [ 2909.279670] Key type .llcrypt unregistered [ 2909.285935] Key type ._llcrypt unregistered [ 2947.050875] Key type ._llcrypt registered [ 2947.054682] Key type .llcrypt registered [ 2947.189774] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 2949.016180] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2949.055341] alg: No test for adler32 (adler32-zlib) [ 2950.499948] Lustre: Lustre: Build Version: 2.17.56_2_gaec046b [ 2950.958688] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 2952.783349] Key type lgssc registered [ 2955.384041] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2969.828836] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 04:17:44 (1785399464) [ 2978.212711] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2978.248208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2980.205022] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2987.362809] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 2989.542088] LustreError: 68950:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2989.564040] LustreError: 68950:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2994.671679] LustreError: 68949:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2999.789393] LustreError: 68950:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3000.720977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3001.215773] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3009.703371] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3027.529551] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3036.055839] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3045.338715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3045.908579] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3049.996543] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 3056.494217] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3067.080493] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3075.613072] Lustre: Failing over lustre-MDT0000 [ 3076.528887] Lustre: server umount lustre-MDT0000 complete [ 3078.139599] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3078.156458] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3078.193664] LustreError: 68949:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3080.674547] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3080.700520] Lustre: Skipped 1 previous similar message [ 3085.697034] LustreError: 68945:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3085.710512] LustreError: 68945:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3091.946859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3092.329789] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3092.737955] Lustre: *** cfs_fail_loc=135, val=0*** [ 3093.016764] LustreError: 72040:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 3093.037350] LustreError: 72040:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.113@tcp: cfg command failed: rc = -2 [ 3093.063334] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 3093.063334] [ 3093.069926] LustreError: MGC192.168.202.113@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3093.116085] LustreError: 72028:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3093.158631] LustreError: 72028:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3093.197050] LustreError: 72028:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3093.212665] LustreError: 72028:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 3093.357961] Lustre: server umount lustre-MDT0000 complete [ 3093.359493] LustreError: 72028:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3095.938323] LustreError: 71249:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3095.972779] LustreError: 71249:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 3103.107553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3103.351938] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3103.368692] LustreError: 68941:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3103.845349] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3103.941369] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3106.198434] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3108.895589] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3108.961422] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 3109.053051] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 3112.772432] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3128.802479] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3128.810130] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3128.821730] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3131.220793] Lustre: server umount lustre-OST0000 complete [ 3139.552280] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3139.591629] Lustre: Skipped 1 previous similar message [ 3139.610415] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3139.613128] Lustre: Skipped 2 previous similar messages [ 3142.783685] Lustre: server umount lustre-MDT0000 complete [ 3144.675221] LustreError: 69797:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3144.709679] LustreError: 69797:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 3147.320023] LustreError: 68929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785399644 with bad export cookie 13602317312090293311 [ 3147.336438] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3147.339432] LustreError: 68929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3147.753787] Lustre: server umount lustre-MDT0001 complete [ 3163.770648] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 3169.674644] Key type lgssc unregistered [ 3170.479700] LNet: 73824:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3170.522487] LNetError: 73824:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3171.567211] LNet: Removed LNI 192.168.202.113@tcp [ 3173.856079] Key type .llcrypt unregistered [ 3173.860905] Key type ._llcrypt unregistered [ 3207.900258] Key type ._llcrypt registered [ 3207.905168] Key type .llcrypt registered [ 3208.223179] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 3209.348245] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3209.390709] alg: No test for adler32 (adler32-zlib) [ 3210.537328] Lustre: Lustre: Build Version: 2.17.56_2_gaec046b [ 3210.893977] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 3212.599166] Key type lgssc registered [ 3213.891634] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3225.608740] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3225.674903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3227.933426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3237.604924] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3238.368627] LustreError: 75243:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3238.381425] LustreError: 75243:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3243.488792] LustreError: 75244:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3248.616487] LustreError: 75243:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3250.580640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3251.509343] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3258.353621] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3270.713219] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3279.430299] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3291.069930] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3291.878017] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3298.117956] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 3303.075742] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3314.336001] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3321.709623] Lustre: Failing over lustre-MDT0000 [ 3322.666501] Lustre: server umount lustre-MDT0000 complete [ 3323.372315] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3323.382516] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3323.396994] LustreError: 75244:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3323.882483] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3323.913339] Lustre: Skipped 1 previous similar message [ 3328.992533] LustreError: 75243:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3329.020258] LustreError: 75243:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 3334.416522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3334.678890] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3334.956647] Lustre: *** cfs_fail_loc=a05, val=0*** [ 3335.199841] LustreError: 78335:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 3335.206701] LustreError: 78335:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.113@tcp: cfg command failed: rc = -9 [ 3335.222753] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 3335.222753] [ 3335.240910] LustreError: MGC192.168.202.113@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3335.253855] LustreError: 78323:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 3335.260279] LustreError: 78323:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 3335.272375] LustreError: 78323:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3335.279913] LustreError: 78323:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 3335.320732] Lustre: server umount lustre-MDT0000 complete [ 3335.325198] LustreError: 78323:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 3339.234035] LustreError: 75243:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3339.258885] LustreError: 75243:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 3343.386755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3343.517286] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3343.527087] LustreError: 75235:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3343.929588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3344.013142] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3346.754332] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3348.993840] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3349.042920] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 3349.117583] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 3350.635545] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3362.785245] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3362.787558] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3362.799663] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3364.333616] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3364.342347] Lustre: Skipped 1 previous similar message [ 3368.582121] Lustre: server umount lustre-OST0000 complete [ 3374.565221] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3374.566830] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3374.606489] Lustre: Skipped 1 previous similar message [ 3374.621784] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3374.639228] Lustre: Skipped 1 previous similar message [ 3379.687915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3379.706122] Lustre: Skipped 1 previous similar message [ 3384.807908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3384.813308] Lustre: Skipped 1 previous similar message [ 3388.384873] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3388.778241] Lustre: server umount lustre-MDT0000 complete [ 3388.915356] LustreError: 78305:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3388.948495] LustreError: 78305:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3393.776426] LustreError: 75224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785399891 with bad export cookie 8469147393894714004 [ 3393.782058] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3393.787610] LustreError: 75224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3394.141322] Lustre: server umount lustre-MDT0001 complete [ 3414.391193] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 3420.336370] Key type lgssc unregistered [ 3421.072647] LNet: 80137:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3421.086908] LNetError: 80137:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3422.126351] LNet: Removed LNI 192.168.202.113@tcp [ 3423.997892] Key type .llcrypt unregistered [ 3424.002853] Key type ._llcrypt unregistered [ 3449.638747] Key type ._llcrypt registered [ 3449.640740] Key type .llcrypt registered [ 3449.796880] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 3450.152406] Key type .llcrypt unregistered [ 3450.158368] Key type ._llcrypt unregistered [ 3483.348448] Key type ._llcrypt registered [ 3483.352429] Key type .llcrypt registered [ 3483.550079] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 3485.709343] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3485.798499] alg: No test for adler32 (adler32-zlib) [ 3487.031405] Lustre: Lustre: Build Version: 2.17.56_2_gaec046b [ 3487.461798] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 3489.375143] Key type lgssc registered [ 3492.115832] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3504.424884] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3504.465661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3506.405601] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3512.852331] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3515.893555] LustreError: 82049:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3515.914212] LustreError: 82049:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3521.006033] LustreError: 82048:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3525.964057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3526.117026] LustreError: 82049:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3526.722620] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3537.120543] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3554.443816] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3566.010891] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3573.419173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3573.986287] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3576.053990] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 3582.355653] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3593.409055] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3612.132259] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3612.133383] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3612.182983] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3612.198315] Lustre: Skipped 1 previous similar message [ 3616.193248] Lustre: server umount lustre-OST0000 complete [ 3622.369553] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3622.408401] Lustre: Skipped 1 previous similar message [ 3622.424055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3627.490746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3627.493014] Lustre: Skipped 1 previous similar message [ 3631.587172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3631.602664] Lustre: Skipped 1 previous similar message [ 3636.191127] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3636.340906] Lustre: server umount lustre-MDT0000 complete [ 3636.707553] LustreError: 83150:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3636.727608] LustreError: 83150:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3640.757488] LustreError: 82030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785400138 with bad export cookie 6915514463372204583 [ 3640.763035] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3640.766480] LustreError: 82030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3641.123070] Lustre: server umount lustre-MDT0001 complete [ 3664.567269] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 3669.568642] Key type lgssc unregistered [ 3669.921464] LNet: 85637:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3669.923687] LNetError: 85637:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3669.936269] LNet: Removed LNI 192.168.202.113@tcp [ 3670.820906] Key type .llcrypt unregistered [ 3670.825567] Key type ._llcrypt unregistered [ 3701.064111] Key type ._llcrypt registered [ 3701.065650] Key type .llcrypt registered [ 3701.210777] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 3702.260949] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3702.375419] alg: No test for adler32 (adler32-zlib) [ 3703.604625] Lustre: Lustre: Build Version: 2.17.56_2_gaec046b [ 3704.043986] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 3705.791238] Key type lgssc registered [ 3707.685339] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3721.373927] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3721.419212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3723.663580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3733.998933] LustreError: 87050:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3734.013412] LustreError: 87050:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3734.629492] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3739.120836] LustreError: 87051:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3739.151557] LustreError: 87051:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3744.227970] LustreError: 87050:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3748.629239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3749.149032] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3756.158201] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3768.746472] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3781.956294] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3795.141527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3795.381149] LustreError: 88875:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3795.863975] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3801.073051] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 3804.757735] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3816.982600] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3826.657153] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3826.662552] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3826.696902] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3827.502861] Lustre: server umount lustre-OST0000 complete [ 3834.864530] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3834.887010] Lustre: Skipped 1 previous similar message [ 3834.895769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3834.922391] Lustre: Skipped 1 previous similar message [ 3838.692975] Lustre: server umount lustre-MDT0000 complete [ 3839.980959] LustreError: 87047:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3840.019711] LustreError: 87047:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3843.949707] LustreError: 87032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785400341 with bad export cookie 8902017581992848766 [ 3844.000066] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3854.596872] Lustre: server umount lustre-MDT0001 complete [ 3874.717623] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 3878.723234] Key type lgssc unregistered [ 3878.954540] LNet: 90612:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3878.961575] LNetError: 90612:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3878.982452] LNet: Removed LNI 192.168.202.113@tcp [ 3880.635789] Key type .llcrypt unregistered [ 3880.637105] Key type ._llcrypt unregistered [ 3917.756396] Key type ._llcrypt registered [ 3917.764268] Key type .llcrypt registered [ 3918.117712] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 3920.616793] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3920.695216] alg: No test for adler32 (adler32-zlib) [ 3922.543920] Lustre: Lustre: Build Version: 2.17.56_2_gaec046b [ 3923.442368] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 3925.576636] Key type lgssc registered [ 3927.840131] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3939.995639] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3940.049138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3941.783864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3948.134763] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3952.107958] LustreError: 92028:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3952.130620] LustreError: 92028:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3957.218447] LustreError: 92029:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3960.862805] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3961.651047] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3967.666552] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 3979.442197] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3988.669153] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3998.151704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3998.787210] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4007.797434] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4012.208624] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 4020.098824] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4038.123159] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4038.123650] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4038.128593] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4038.137526] Lustre: Skipped 1 previous similar message [ 4043.234666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4043.240027] Lustre: Skipped 2 previous similar messages [ 4044.497877] Lustre: server umount lustre-OST0000 complete [ 4053.478973] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4053.510191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4053.522866] Lustre: Skipped 1 previous similar message [ 4056.577235] Lustre: server umount lustre-MDT0000 complete [ 4058.594087] LustreError: 92024:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4058.608228] LustreError: 92024:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4061.300156] LustreError: 92010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785400558 with bad export cookie 1740194687793915758 [ 4061.313313] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4061.317933] LustreError: 92010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4062.268468] Lustre: server umount lustre-MDT0001 complete [ 4076.848667] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 4079.931656] Key type lgssc unregistered [ 4080.349535] LNet: 95589:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4080.353375] LNetError: 95589:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4081.386113] LNet: Removed LNI 192.168.202.113@tcp [ 4083.383768] Key type .llcrypt unregistered [ 4083.387392] Key type ._llcrypt unregistered [ 4120.173592] Key type ._llcrypt registered [ 4120.179152] Key type .llcrypt registered [ 4120.534384] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 4122.672015] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4122.788823] alg: No test for adler32 (adler32-zlib) [ 4124.312163] Lustre: Lustre: Build Version: 2.17.56_2_gaec046b [ 4124.944496] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 4126.895155] Key type lgssc registered [ 4129.526751] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4143.551178] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4143.610437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4159.328549] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4164.447470] LustreError: 96983:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.113@tcp: failed processing log, type 4: rc = -110 [ 4191.135602] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4200.350740] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4211.408642] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4218.590739] Lustre: server umount lustre-OST0000 complete [ 4230.217099] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 04:38:44 (1785400724) [ 4238.014397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4238.615995] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4243.940274] LustreError: 98624:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4243.973830] LustreError: 98624:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4244.560169] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4249.059921] LustreError: 98623:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4254.177187] LustreError: 98624:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4256.499540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4257.100630] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4263.366463] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4274.315544] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4281.452233] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4289.401214] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4290.096160] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4299.588980] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4308.526192] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 4310.646589] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4323.811724] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4323.827264] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4323.828828] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4328.931062] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4328.946976] Lustre: Skipped 2 previous similar messages [ 4330.104805] Lustre: server umount lustre-OST0000 complete [ 4339.177074] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4339.185102] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4339.197644] Lustre: Skipped 1 previous similar message [ 4339.209882] LustreError: Skipped 1 previous similar message [ 4339.211799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4344.289684] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4344.295533] Lustre: Skipped 2 previous similar messages [ 4349.407356] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4349.409979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4349.437053] Lustre: Skipped 1 previous similar message [ 4350.411924] Lustre: server umount lustre-MDT0000 complete [ 4353.508172] LustreError: 98623:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4353.546903] LustreError: 98623:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4356.004575] LustreError: 98605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785400853 with bad export cookie 804052835307652683 [ 4356.021214] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4356.024108] LustreError: 98605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4357.138951] Lustre: server umount lustre-MDT0001 complete [ 4375.016528] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 4379.412842] Key type lgssc unregistered [ 4379.861955] LNet: 102319:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4379.884724] LNetError: 102319:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4380.903892] LNet: Removed LNI 192.168.202.113@tcp [ 4382.316533] Key type .llcrypt unregistered [ 4382.319792] Key type ._llcrypt unregistered [ 4439.202905] Key type ._llcrypt registered [ 4439.206680] Key type .llcrypt registered [ 4439.496603] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 4441.800039] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4442.225323] alg: No test for adler32 (adler32-zlib) [ 4443.532976] Lustre: Lustre: Build Version: 2.17.56_2_gaec046b [ 4443.996814] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 4445.775314] Key type lgssc registered [ 4448.151407] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4464.829604] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4464.900573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4467.408421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4475.814873] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4476.965497] LustreError: 104510:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4477.022051] LustreError: 104510:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4482.016786] LustreError: 104511:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4487.150105] LustreError: 104510:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4487.255385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4487.780844] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4496.515837] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4510.830742] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4524.818114] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4536.469455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4537.058665] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4542.456351] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 4546.249394] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4557.344346] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4573.161166] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4573.165873] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4573.183861] Lustre: Skipped 1 previous similar message [ 4573.211972] Lustre: Skipped 1 previous similar message [ 4577.790907] Lustre: server umount lustre-OST0000 complete [ 4588.513871] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4588.550200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4588.553636] Lustre: Skipped 1 previous similar message [ 4592.588641] Lustre: server umount lustre-MDT0000 complete [ 4592.865980] LustreError: 104510:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4592.922438] LustreError: 104510:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4598.269335] LustreError: 104510:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4599.307028] LustreError: 104491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785401096 with bad export cookie 14902071347330556339 [ 4599.335919] LustreError: 104491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4599.338235] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4600.214416] Lustre: server umount lustre-MDT0001 complete [ 4618.864931] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 4625.038971] Key type lgssc unregistered [ 4625.713809] LNet: 108074:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4625.732445] LNetError: 108074:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4626.796855] LNet: Removed LNI 192.168.202.113@tcp [ 4628.904926] Key type .llcrypt unregistered [ 4628.906354] Key type ._llcrypt unregistered [ 4656.789983] Key type ._llcrypt registered [ 4656.792260] Key type .llcrypt registered [ 4657.025559] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 4659.351525] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4659.510795] alg: No test for adler32 (adler32-zlib) [ 4660.729332] Lustre: Lustre: Build Version: 2.17.56_2_gaec046b [ 4661.050075] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 4662.807915] Key type lgssc registered [ 4665.088116] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4676.487890] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4676.526493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4692.196637] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4697.375348] LustreError: 109319:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.113@tcp: failed processing log, type 4: rc = -110 [ 4724.127206] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4733.532771] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4746.415166] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4759.901577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4760.824398] LustreError: MGC192.168.202.113@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 4760.884894] LustreError: 110347:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 4760.911608] LustreError: 110347:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 4760.933738] LustreError: 110347:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 4761.231459] Lustre: server umount lustre-MDT0000 complete [ 4761.245407] LustreError: 110347:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4782.049800] Lustre: 108656:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785401263/real 1785401263] req@00000000d3f7edb6 x1872128804654720/t0(0) o400->MGC192.168.202.113@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1785401279 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4782.098294] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4782.119551] LustreError: 109331:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 4802.747082] Lustre: server umount lustre-OST0000 complete [ 4820.597850] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_hostid [ 4840.249783] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 4897.925882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4898.186572] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4898.283867] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4898.380194] Lustre: lustre-MDT0000: new disk, initializing [ 4898.459374] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4898.481236] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4904.413791] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4918.333159] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4918.612234] Lustre: 113169:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4918.741373] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4918.744215] Lustre: Skipped 1 previous similar message [ 4918.833652] Lustre: lustre-MDT0001: new disk, initializing [ 4919.046973] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4919.130994] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4919.155056] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4925.684277] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4940.268870] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4949.132628] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4958.481824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4958.811571] Lustre: lustre-OST0000: new disk, initializing [ 4958.820517] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4958.829553] Lustre: 115138:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4958.996315] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4967.406367] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 4968.543031] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4968.568807] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4968.792716] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4981.296607] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4987.846118] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4988.323746] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4996.122842] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4996.315971] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4999.648308] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4999.655610] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4999.662142] LustreError: Skipped 1 previous similar message [ 4999.665542] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4999.671507] Lustre: Skipped 1 previous similar message [ 5005.083251] Lustre: server umount lustre-OST0000 complete [ 5009.893566] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5009.916194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5009.919880] Lustre: Skipped 2 previous similar messages [ 5015.009666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5015.015788] Lustre: Skipped 1 previous similar message [ 5020.128308] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5020.136724] Lustre: Skipped 1 previous similar message [ 5024.225244] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5024.660415] Lustre: server umount lustre-MDT0000 complete [ 5025.253182] LustreError: 113175:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5025.281811] LustreError: 113175:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 5029.522432] LustreError: 113161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785401527 with bad export cookie 11952421348016916819 [ 5029.523850] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5029.540127] LustreError: 113161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5030.910502] Lustre: server umount lustre-MDT0001 complete [ 5043.371710] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 04:52:18 (1785401538) [ 5050.563989] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 5091.011521] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_hostid [ 5117.715218] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 5176.541064] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5177.042977] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5177.075653] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5177.188102] Lustre: lustre-MDT0000: new disk, initializing [ 5177.292554] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5177.328072] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5184.869671] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 5198.847256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5198.992919] Lustre: 120074:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5199.040726] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5199.045159] Lustre: Skipped 1 previous similar message [ 5199.120811] Lustre: lustre-MDT0001: new disk, initializing [ 5199.295922] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5199.365020] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5199.387065] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5205.514774] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 5218.312295] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5226.380294] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5235.616233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5236.189217] Lustre: lustre-OST0000: new disk, initializing [ 5236.199543] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5236.209743] Lustre: 122042:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5236.460700] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5245.895015] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 5246.049588] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5246.062038] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5246.193328] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5262.990616] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5272.065898] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5272.470877] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5279.755690] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5280.260807] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5287.416413] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5287.422306] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5287.433649] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5287.443827] Lustre: Skipped 2 previous similar messages [ 5287.473538] LustreError: Skipped 1 previous similar message [ 5289.708714] Lustre: server umount lustre-OST0000 complete [ 5296.608572] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5296.623691] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5296.655183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5296.664678] Lustre: Skipped 1 previous similar message [ 5302.426890] Lustre: server umount lustre-MDT0000 complete [ 5302.760040] LustreError: 120088:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5302.805659] LustreError: 120088:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5307.874670] LustreError: 120084:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5307.913877] LustreError: 120084:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5309.387421] LustreError: 120068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785401806 with bad export cookie 11952421348016917701 [ 5309.404437] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5309.409265] LustreError: 120068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5310.325271] Lustre: server umount lustre-MDT0001 complete [ 5325.059748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5325.578087] LustreError: 124265:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5325.600906] LustreError: 124265:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5325.655994] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5330.915237] LustreError: 124266:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5332.824614] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 5341.177400] LustreError: 124265:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5341.215229] LustreError: 124265:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5346.207160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5355.767845] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 5368.489901] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5378.380214] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5390.879786] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5391.640258] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5391.656162] Lustre: Skipped 1 previous similar message [ 5401.830871] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 5412.926328] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5422.073833] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 5431.778552] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5431.793272] Lustre: Skipped 1 previous similar message [ 5431.899400] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5431.943970] Lustre: Skipped 2 previous similar messages [ 5436.066338] Lustre: server umount lustre-OST0000 complete [ 5442.035456] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5442.047624] Lustre: Skipped 2 previous similar messages [ 5456.352449] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5456.684711] Lustre: server umount lustre-MDT0000 complete [ 5457.378174] LustreError: 124265:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5457.391702] LustreError: 124265:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5462.993206] LustreError: 124245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785401960 with bad export cookie 11952421348016918583 [ 5463.012195] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5463.015771] LustreError: 124245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5463.865806] Lustre: server umount lustre-MDT0001 complete [ 5482.117371] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 5489.322796] Key type lgssc unregistered [ 5490.097921] LNet: 127903:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5490.158101] LNetError: 127903:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5490.191950] LNet: Removed LNI 192.168.202.113@tcp [ 5492.193944] Key type .llcrypt unregistered [ 5492.201090] Key type ._llcrypt unregistered [ 5526.283637] Key type ._llcrypt registered [ 5526.289876] Key type .llcrypt registered [ 5526.521213] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_hostid [ 5550.733136] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 5553.033853] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5553.206314] alg: No test for adler32 (adler32-zlib) [ 5555.170193] Lustre: Lustre: Build Version: 2.17.56_2_gaec046b [ 5555.697396] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 5557.543198] Key type lgssc registered [ 5559.798833] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5613.002167] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5613.080605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5614.726453] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5614.825432] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5615.010935] Lustre: lustre-MDT0000: new disk, initializing [ 5615.168191] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5615.220804] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5621.782804] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 5637.299085] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5637.567647] Lustre: 131076:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5637.710836] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5637.719181] Lustre: Skipped 1 previous similar message [ 5637.915185] Lustre: lustre-MDT0001: new disk, initializing [ 5638.102477] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5638.173200] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5638.190395] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5643.583747] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 5656.794252] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5666.373251] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5678.627581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5679.002172] Lustre: lustre-OST0000: new disk, initializing [ 5679.007891] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5679.015640] Lustre: 133050:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5679.129592] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5687.687178] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 5688.416878] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5688.423369] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5688.597301] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5702.757970] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5710.762472] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5711.203863] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5718.271668] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5718.463856] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5724.640705] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5724.643580] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5724.657393] LustreError: Skipped 1 previous similar message [ 5724.660904] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5724.670792] Lustre: Skipped 1 previous similar message [ 5724.691368] Lustre: Skipped 1 previous similar message [ 5727.735536] Lustre: server umount lustre-OST0000 complete [ 5734.889026] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5734.908972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5740.000299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5740.026843] Lustre: Skipped 2 previous similar messages [ 5745.122497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5745.128675] Lustre: Skipped 1 previous similar message [ 5747.167345] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5747.413876] Lustre: server umount lustre-MDT0000 complete [ 5752.798713] LustreError: 131873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785402250 with bad export cookie 1026482357074846238 [ 5752.803605] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5752.821588] LustreError: 131873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5753.385450] Lustre: server umount lustre-MDT0001 complete [ 5764.867908] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 05:04:19 (1785402259) [ 5775.194974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5776.037943] LustreError: 135466:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5776.049703] LustreError: 135466:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5776.117209] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5781.504145] LustreError: 135467:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5785.627894] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 5785.634994] LustreError: 135466:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5790.698879] LustreError: 135467:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5795.809660] LustreError: 135466:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5797.093590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5797.378637] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5802.718241] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 5813.920671] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5822.663379] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5827.040420] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5827.043666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5827.067072] Lustre: Skipped 2 previous similar messages [ 5832.170622] LustreError: 135466:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5832.328737] Lustre: server umount lustre-MDT0000 complete [ 5838.042820] LustreError: 135446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785402335 with bad export cookie 1026482357074847120 [ 5838.049892] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5838.061532] LustreError: 135446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5838.633706] Lustre: server umount lustre-MDT0001 complete [ 5852.346396] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 05:05:47 (1785402347) [ 5865.590840] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5881.504512] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5886.624399] LustreError: 137889:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.113@tcp: failed processing log, type 4: rc = -110 [ 5912.352122] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5923.727616] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 5932.454684] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5935.829130] Lustre: server umount lustre-OST0000 complete [ 5951.519974] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 05:07:26 (1785402446) [ 5962.783256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5963.752346] LustreError: 139339:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5963.777860] LustreError: 139339:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5963.882459] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5970.574781] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 5981.852091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5988.538916] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 5999.678354] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6006.957262] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6016.035606] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6016.253446] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6016.257334] Lustre: Skipped 1 previous similar message [ 6025.425846] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 6034.550877] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6043.616878] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6043.619480] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6043.648366] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6043.654411] Lustre: Skipped 1 previous similar message [ 6046.906724] Lustre: server umount lustre-MDT0000 complete [ 6049.249465] LustreError: 139339:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6049.282840] LustreError: 139339:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6051.323769] LustreError: 139319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785402548 with bad export cookie 1026482357074847785 [ 6051.327612] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6051.336938] LustreError: 139319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6051.966414] Lustre: server umount lustre-MDT0001 complete [ 6067.522757] Lustre: server umount lustre-OST0000 complete [ 6079.625597] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 05:09:34 (1785402574) [ 6092.144100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6096.707341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6106.587228] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 6113.777502] LustreError: 142999:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6113.792789] LustreError: 142999:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6117.888590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6125.960586] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 6137.361162] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6148.625328] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6161.739293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6164.682490] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 6176.670991] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 6191.124969] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6197.036680] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6197.434949] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6205.225837] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6205.574537] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6210.019202] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6210.025980] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6210.045465] Lustre: Skipped 2 previous similar messages [ 6210.053754] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6210.063773] Lustre: Skipped 3 previous similar messages [ 6214.573837] Lustre: server umount lustre-OST0000 complete [ 6221.280997] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6221.292033] Lustre: Skipped 2 previous similar messages [ 6227.704779] Lustre: server umount lustre-MDT0000 complete [ 6232.157766] LustreError: 142979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785402729 with bad export cookie 1026482357074848856 [ 6232.182190] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6232.734326] Lustre: server umount lustre-MDT0001 complete [ 6242.550748] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 05:12:17 (1785402737) [ 6252.918034] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6268.642104] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6273.761865] LustreError: 147165:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.113@tcp: failed processing log, type 4: rc = -110 [ 6299.551622] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6299.559118] Lustre: Skipped 2 previous similar messages [ 6309.407944] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 6323.244531] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6333.878632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6334.676846] LustreError: 147190:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6334.700626] LustreError: 147190:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6335.005165] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 6340.971519] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 6351.913617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6359.973820] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 6372.758502] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6384.221056] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6392.916483] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6393.285816] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6404.387538] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6405.216565] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6409.184948] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6409.192422] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6409.210689] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6409.210696] Lustre: Skipped 6 previous similar messages [ 6415.535299] Lustre: server umount lustre-MDT0000 complete [ 6419.875104] LustreError: 147172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785402917 with bad export cookie 1026482357074849878 [ 6419.886672] LustreError: 147172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6419.889756] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6420.516806] Lustre: server umount lustre-MDT0001 complete [ 6435.726121] Lustre: server umount lustre-OST0000 complete [ 6447.438447] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 05:15:42 (1785402942) [ 6456.606233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6472.352268] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6477.471559] LustreError: 151368:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.113@tcp: failed processing log, type 4: rc = -110 [ 6512.280490] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 6526.196586] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6535.403921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6536.289028] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 6542.552836] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 6553.606714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6561.419098] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 6574.165702] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6584.592586] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6594.871821] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6595.171991] Lustre: lustre-OST0001: new disk, initializing [ 6595.179618] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6595.192301] Lustre: 154243:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6595.359281] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6595.367729] Lustre: Skipped 5 previous similar messages [ 6603.355762] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6603.378316] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6603.572815] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6603.974080] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 6617.518485] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6624.210496] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6624.549624] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6630.952175] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6631.237374] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6636.002981] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6636.018856] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6636.042731] Lustre: Skipped 2 previous similar messages [ 6636.050869] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6636.062754] Lustre: Skipped 5 previous similar messages [ 6640.888763] Lustre: server umount lustre-OST0000 complete [ 6644.708380] LustreError: 151383:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6644.754392] LustreError: 151383:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 21 previous similar messages [ 6649.826403] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6649.837382] LustreError: Skipped 2 previous similar messages [ 6654.530204] Lustre: server umount lustre-OST0001 complete [ 6666.686042] Lustre: server umount lustre-MDT0000 complete [ 6671.327613] LustreError: 151374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785403168 with bad export cookie 1026482357074850830 [ 6671.362559] LustreError: 151374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6671.371292] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6672.024923] Lustre: server umount lustre-MDT0001 complete [ 6705.942182] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 05:20:01 (1785403201) [ 6708.916973] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 6711.002928] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 6712.833800] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 05:20:08 (1785403208) [ 6721.344803] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6721.709681] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6721.782167] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6728.694573] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 6740.801970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6740.872618] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6740.898200] Lustre: 157809:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6745.634529] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 6756.265482] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6763.142842] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6771.127193] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6771.270655] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6780.205761] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 6780.407514] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 6790.062547] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6796.928583] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6797.184697] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6803.763847] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6804.257869] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6811.104135] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6811.107393] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6811.117777] LustreError: Skipped 2 previous similar messages [ 6811.121044] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6811.121050] Lustre: Skipped 8 previous similar messages [ 6811.193194] Lustre: Skipped 6 previous similar messages [ 6821.343184] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6821.637766] Lustre: server umount lustre-OST0000 complete [ 6850.405624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6851.879792] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6851.920060] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6851.930333] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6851.946917] Lustre: Skipped 1 previous similar message [ 6860.924209] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 6871.173963] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6878.636521] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6879.052854] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6886.592000] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6887.112023] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6896.385766] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8bcd100a2000.ost_server_uuid 50 [ 6898.902951] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8bcd100a2000.ost_server_uuid in IDLE state after 0 sec [ 6899.274722] LustreError: 157816:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.202.13@tcp) returned error from blocking AST (req@0000000098562e23 x1872129743445760 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 000000000342fa44/0xe3ecc43e5a6fd68 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.13@tcp remote: 0x253f9ffc4e9a64cc expref: 5 pid: 159433 timeout: 6999 lvb_type: 0 lru_score: 0 lru_type: 0 [ 6899.334696] LustreError: lustre-MDT0000: A client on nid 192.168.202.13@tcp was evicted due to a lock blocking callback time out: rc -107 [ 6899.346250] LustreError: 157804:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.202.13@tcp ns: mdt-lustre-MDT0000_UUID lock: 000000000342fa44/0xe3ecc43e5a6fd68 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.13@tcp remote: 0x253f9ffc4e9a64cc expref: 6 pid: 159433 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 6908.384071] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6908.397103] LustreError: Skipped 1 previous similar message [ 6911.654516] Lustre: server umount lustre-OST0000 complete [ 6924.757859] Lustre: server umount lustre-MDT0000 complete [ 6929.798698] LustreError: 157799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785403427 with bad export cookie 1026482357074852013 [ 6929.823291] LustreError: 157799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6929.830176] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6944.223249] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6944.821821] Lustre: server umount lustre-MDT0001 complete [ 6960.818469] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 6965.434319] Key type lgssc unregistered [ 6965.896647] LNet: 163931:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6965.903851] LNetError: 163931:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6965.920512] LNet: Removed LNI 192.168.202.113@tcp [ 6967.059476] Key type .llcrypt unregistered [ 6967.069925] Key type ._llcrypt unregistered [ 6994.063272] Key type ._llcrypt registered [ 6994.068720] Key type .llcrypt registered [ 6994.182923] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 6995.746617] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6995.882431] alg: No test for adler32 (adler32-zlib) [ 6996.908819] Lustre: Lustre: Build Version: 2.17.56_2_gaec046b [ 6997.072464] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 6998.727531] Key type lgssc registered [ 7000.641494] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7013.250427] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7013.263619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7014.931026] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7021.566129] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 7025.134618] LustreError: 165349:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7025.166509] LustreError: 165349:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7030.246661] LustreError: 165350:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7033.684537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7034.560917] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7040.505435] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 7050.604581] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7058.675793] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7068.204983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7069.041386] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7077.239856] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 7085.497928] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 7088.128901] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7096.552351] Lustre: Failing over lustre-MDT0000 [ 7097.642326] Lustre: server umount lustre-MDT0000 complete [ 7100.898508] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7100.900116] LustreError: 167396:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7100.930199] Lustre: Skipped 2 previous similar messages [ 7100.960306] LustreError: 167396:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7106.026179] LustreError: 165345:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7106.065018] LustreError: 165345:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7110.559680] LustreError: 165328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.13@tcp arrived at 1785403608 with bad export cookie 15963310532048769111 [ 7110.572022] LustreError: 165328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7111.137930] LustreError: 167844:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7111.164045] LustreError: 167844:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7117.280090] Lustre: 164657:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785403598/real 1785403598] req@000000003583ae7f x1872131254156288/t0(0) o400->MGC192.168.202.113@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1785403614 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7117.300106] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7121.086073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7122.402204] LustreError: 167396:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7122.437708] LustreError: 167396:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 7127.031992] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7127.147455] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7132.131698] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7132.156684] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7134.341934] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 7137.650559] Lustre: lustre-MDT0000: Denying connection for new client 26f61d1b-2796-4b7e-b94b-53be8feead8f (at 192.168.202.13@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:04 [ 7147.491732] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7147.499902] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7147.510862] Lustre: Skipped 1 previous similar message [ 7147.527580] Lustre: Skipped 1 previous similar message [ 7152.612190] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7152.630755] Lustre: Skipped 1 previous similar message [ 7157.736474] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7157.755835] Lustre: Skipped 1 previous similar message [ 7161.312673] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7161.720913] Lustre: server umount lustre-OST0000 complete [ 7167.238766] LustreError: 169088:0:(ldlm_lib.c:3003:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 7167.250848] Lustre: 168435:0:(ldlm_lib.c:2403:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 7167.265366] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 7167.294957] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 7167.307533] LustreError: 168435:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000006652e1e x1872131254178688/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7167.334447] LustreError: 168435:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 7167.354275] LustreError: 168435:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 7167.372210] Lustre: lustre-MDT0000: Recovery over after 0:35, of 2 clients 0 recovered and 2 were evicted. [ 7167.974056] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7167.997036] LustreError: 165344:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7168.018951] LustreError: 165344:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7168.281112] Lustre: server umount lustre-MDT0000 complete [ 7174.213594] LustreError: 166084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785403671 with bad export cookie 15963310532048769209 [ 7174.214963] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7174.226715] LustreError: 166084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7174.236218] LustreError: 164655:0:(import.c:707:ptlrpc_connect_import_locked()) can't connect to a closed import [ 7181.665854] Lustre: server umount lustre-MDT0001 complete [ 7201.495772] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 7205.828796] Key type lgssc unregistered [ 7206.158238] LNet: 169873:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7206.169556] LNetError: 169873:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7206.198568] LNet: Removed LNI 192.168.202.113@tcp [ 7207.592752] Key type .llcrypt unregistered [ 7207.594156] Key type ._llcrypt unregistered [ 7242.336970] Key type ._llcrypt registered [ 7242.339247] Key type .llcrypt registered [ 7242.597616] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 7245.403629] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7245.749549] alg: No test for adler32 (adler32-zlib) [ 7246.962918] Lustre: Lustre: Build Version: 2.17.56_2_gaec046b [ 7247.375269] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 7249.199169] Key type lgssc registered [ 7250.671243] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7261.753537] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7261.794897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7263.602730] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7269.814448] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 7273.953135] LustreError: 171304:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7273.991830] LustreError: 171304:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7279.073878] LustreError: 171303:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7281.882390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7282.478632] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7289.955539] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 7302.660967] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7315.354894] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7324.768132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7325.515976] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7333.820172] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 7335.790877] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 7349.462451] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7359.460993] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7359.479808] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7359.512418] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7359.517078] Lustre: Skipped 1 previous similar message [ 7359.577643] Lustre: Skipped 1 previous similar message [ 7364.582843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7366.151539] Lustre: server umount lustre-OST0000 complete [ 7374.820312] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7374.822731] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7374.834156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7374.834163] Lustre: Skipped 1 previous similar message [ 7379.656725] Lustre: server umount lustre-MDT0000 complete [ 7379.944963] LustreError: 171298:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7379.971524] LustreError: 171298:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7384.905085] LustreError: 171284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785403882 with bad export cookie 11239642099004453709 [ 7384.905800] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7384.912854] LustreError: 171284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7385.845821] Lustre: server umount lustre-MDT0001 complete [ 7411.815749] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 7417.484741] Key type lgssc unregistered [ 7417.741432] LNet: 174866:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7417.744572] LNetError: 174866:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7417.765659] LNet: Removed LNI 192.168.202.113@tcp [ 7419.223828] Key type .llcrypt unregistered [ 7419.229112] Key type ._llcrypt unregistered [ 7476.598045] Key type ._llcrypt registered [ 7476.610837] Key type .llcrypt registered [ 7476.927257] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 7480.140494] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7480.208833] alg: No test for adler32 (adler32-zlib) [ 7481.321883] Lustre: Lustre: Build Version: 2.17.56_2_gaec046b [ 7481.526904] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 7483.184073] Key type lgssc registered [ 7484.972396] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7496.615601] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7496.644840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7498.289418] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7504.004408] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 7507.496835] LustreError: 176572:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7507.528609] LustreError: 176572:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7512.544913] LustreError: 176573:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7515.854862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7516.436193] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7521.420859] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 7534.323636] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7542.778283] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7552.282617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7553.038329] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7562.178811] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 7568.463744] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 7573.527832] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7591.905735] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7591.910132] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7591.917967] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7593.956879] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7593.960332] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7593.967320] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7595.992626] Lustre: server umount lustre-OST0000 complete [ 7604.202590] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7604.226864] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7604.240095] Lustre: Skipped 2 previous similar messages [ 7609.316811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7609.341986] Lustre: Skipped 1 previous similar message [ 7610.692396] Lustre: server umount lustre-MDT0000 complete [ 7614.452119] LustreError: 176573:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7614.519749] LustreError: 176573:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7616.382732] LustreError: 176553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785404113 with bad export cookie 17052736855478482534 [ 7616.402090] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7617.377976] Lustre: server umount lustre-MDT0001 complete [ 7634.690445] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 7639.470935] Key type lgssc unregistered [ 7639.966515] LNet: 180133:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7639.975118] LNetError: 180133:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7640.998068] LNet: Removed LNI 192.168.202.113@tcp [ 7642.554070] Key type .llcrypt unregistered [ 7642.556542] Key type ._llcrypt unregistered [ 7672.995825] Key type ._llcrypt registered [ 7672.997330] Key type .llcrypt registered [ 7673.070648] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 7674.241590] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7674.666414] alg: No test for adler32 (adler32-zlib) [ 7676.281556] Lustre: Lustre: Build Version: 2.17.56_2_gaec046b [ 7676.468559] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 7678.159162] Key type lgssc registered [ 7679.663918] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7692.865059] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7692.906075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7694.391633] Lustre: *** cfs_fail_loc=135, val=0*** [ 7694.507260] LustreError: 181660:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 7694.518451] LustreError: 181660:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.113@tcp: cfg command failed: rc = -2 [ 7694.530048] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 7694.530048] [ 7694.543707] LustreError: MGC192.168.202.113@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 7694.556961] LustreError: 181621:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 7694.575487] LustreError: 181621:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 7694.586021] LustreError: 181621:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 7694.591486] LustreError: 181621:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 7694.677804] Lustre: server umount lustre-MDT0000 complete [ 7694.685058] LustreError: 181621:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 7703.294435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7703.967102] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7709.983992] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 7714.275431] LustreError: 182032:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7714.311782] LustreError: 182032:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7719.395668] LustreError: 182031:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7722.435381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7723.238672] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7730.184329] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 7740.348680] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7748.154510] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7753.696394] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7753.706804] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7753.729053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7755.235532] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7755.236907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7755.259633] Lustre: Skipped 1 previous similar message [ 7757.712173] Lustre: server umount lustre-MDT0000 complete [ 7760.352984] LustreError: 182759:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7760.373875] LustreError: 182759:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7763.035001] LustreError: 182013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785404260 with bad export cookie 4709534934629701281 [ 7763.048140] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7763.728718] Lustre: server umount lustre-MDT0001 complete [ 7780.861375] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 7784.140990] Key type lgssc unregistered [ 7784.353523] LNet: 184355:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7784.359202] LNetError: 184355:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7784.398480] LNet: Removed LNI 192.168.202.113@tcp [ 7785.332870] Key type .llcrypt unregistered [ 7785.336847] Key type ._llcrypt unregistered [ 7820.918193] Key type ._llcrypt registered [ 7820.928706] Key type .llcrypt registered [ 7821.140912] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 7821.597278] Key type .llcrypt unregistered [ 7821.601160] Key type ._llcrypt unregistered [ 7853.455615] Key type ._llcrypt registered [ 7853.458501] Key type .llcrypt registered [ 7853.641796] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 7855.370593] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7855.410308] alg: No test for adler32 (adler32-zlib) [ 7856.797860] Lustre: Lustre: Build Version: 2.17.56_2_gaec046b [ 7857.355830] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 7859.271191] Key type lgssc registered [ 7861.626234] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7875.836326] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7875.886668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7891.809377] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7896.927607] LustreError: 186493:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.113@tcp: failed processing log, type 4: rc = -110 [ 7923.617250] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7933.569609] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 7944.460750] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7954.604388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7955.604724] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7955.745398] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 7963.106853] LustreError: 187535:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7963.168693] LustreError: 187535:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 7964.568743] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 7968.227323] LustreError: 187533:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7968.268089] LustreError: 187533:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7973.349636] LustreError: 187534:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7973.380614] LustreError: 187534:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7978.470769] LustreError: 187533:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7978.529016] LustreError: 187533:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7980.815533] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7981.472768] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7990.019796] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 8000.263866] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8009.012532] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8014.413401] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 8027.112234] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8027.119757] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8027.150289] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8027.620328] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8031.600860] Lustre: server umount lustre-OST0000 complete [ 8037.857057] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8037.890952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8037.909337] Lustre: Skipped 2 previous similar messages [ 8042.976988] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8048.103486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8048.124324] Lustre: Skipped 2 previous similar messages [ 8051.173275] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8051.893491] Lustre: server umount lustre-MDT0000 complete [ 8054.242245] LustreError: 187539:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8054.253028] LustreError: 187539:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8059.473191] LustreError: 186501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785404556 with bad export cookie 15123071477628715275 [ 8059.501054] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8060.760381] Lustre: server umount lustre-MDT0001 complete [ 8081.530858] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 8092.803225] Key type lgssc unregistered [ 8093.451937] LNet: 190420:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8093.457128] LNetError: 190420:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8093.512905] LNet: Removed LNI 192.168.202.113@tcp [ 8098.983067] Key type .llcrypt unregistered [ 8098.986530] Key type ._llcrypt unregistered [ 8141.245408] Key type ._llcrypt registered [ 8141.246812] Key type .llcrypt registered [ 8141.326133] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 8143.662768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8143.788980] alg: No test for adler32 (adler32-zlib) [ 8145.573406] Lustre: Lustre: Build Version: 2.17.56_2_gaec046b [ 8146.079443] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 8147.712075] Key type lgssc registered [ 8150.674331] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8164.879206] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8164.920813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8166.791269] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8173.777150] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 8177.138315] LustreError: 191837:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8177.196130] LustreError: 191837:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8182.242198] LustreError: 191838:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8186.685929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8187.204632] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8194.726587] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 8206.826807] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8218.753521] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8228.492207] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8229.058436] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8237.259317] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 8238.018614] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 8247.629630] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8254.373076] Lustre: Failing over lustre-MDT0000 [ 8255.230294] Lustre: server umount lustre-MDT0000 complete [ 8255.910830] LustreError: 191832:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8258.034043] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8258.056660] Lustre: Skipped 2 previous similar messages [ 8258.069974] LustreError: 191833:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8258.097918] LustreError: 191833:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8263.137371] LustreError: 191837:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8263.156562] LustreError: 191837:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8271.257067] LustreError: 193684:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8271.282561] LustreError: 193684:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 8274.426583] Lustre: 191150:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785404755/real 1785404755] req@000000003a45e716 x1872132458969984/t0(0) o400->MGC192.168.202.113@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1785404771 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8274.461260] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8279.446029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8283.946906] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8284.039996] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8287.645668] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8289.082791] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 8289.266091] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8289.290439] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 8289.342961] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 8295.064637] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 8302.999546] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 8314.336811] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8314.351993] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8314.366047] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8317.353888] Lustre: server umount lustre-OST0000 complete [ 8325.089441] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8325.094434] Lustre: Skipped 1 previous similar message [ 8325.097337] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8325.104488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8325.112244] Lustre: Skipped 2 previous similar messages [ 8329.052305] Lustre: server umount lustre-MDT0000 complete [ 8330.208792] LustreError: 191837:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8330.222498] LustreError: 191837:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 17 previous similar messages [ 8334.622680] LustreError: 193688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785404832 with bad export cookie 11589058773671856268 [ 8334.629491] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8334.632993] LustreError: 193688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8335.454157] Lustre: server umount lustre-MDT0001 complete [ 8355.442286] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 8358.618488] Key type lgssc unregistered [ 8358.877622] LNet: 196708:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8358.880713] LNetError: 196708:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8358.908165] LNet: Removed LNI 192.168.202.113@tcp [ 8359.751704] Key type .llcrypt unregistered [ 8359.753454] Key type ._llcrypt unregistered [ 8393.647220] Key type ._llcrypt registered [ 8393.653267] Key type .llcrypt registered [ 8393.904284] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 8396.372407] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8396.453882] alg: No test for adler32 (adler32-zlib) [ 8398.236850] Lustre: Lustre: Build Version: 2.17.56_2_gaec046b [ 8398.822658] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 8400.679183] Key type lgssc registered [ 8402.664578] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8412.814739] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8412.851149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8414.628679] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8420.955181] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 8424.935980] LustreError: 198124:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8424.965951] LustreError: 198124:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8430.052135] LustreError: 198125:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8433.171972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8433.752774] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8440.480375] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 8450.832925] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8462.183223] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8472.316264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8472.935184] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8481.482625] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 8485.235822] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 8492.258961] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8497.318495] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 8505.221668] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 8524.971678] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 8544.380590] Lustre: server umount lustre-OST0000 complete [ 8551.392720] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8551.398393] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8551.417026] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8551.906062] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8551.916082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8557.028739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8557.039245] Lustre: Skipped 1 previous similar message [ 8560.213500] Lustre: server umount lustre-MDT0000 complete [ 8561.632358] Lustre: 197434:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785405043/real 1785405043] req@00000000a9a0f261 x1872132724288896/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1785405059 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8561.669118] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8561.683220] LustreError: 198852:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8561.707446] LustreError: 198852:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8563.167131] Lustre: 197436:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785405044/real 1785405044] req@00000000e00ec930 x1872132724289664/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1785405060 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8564.656744] LustreError: 198106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785405062 with bad export cookie 18116492394781395922 [ 8564.661144] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8564.707456] LustreError: 198106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8565.969386] Lustre: server umount lustre-MDT0001 complete [ 8580.309315] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 8584.986609] Key type lgssc unregistered [ 8585.380052] LNet: 202026:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8585.387171] LNetError: 202026:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8585.428575] LNet: Removed LNI 192.168.202.113@tcp [ 8586.908260] Key type .llcrypt unregistered [ 8586.913318] Key type ._llcrypt unregistered [ 8614.001527] Key type ._llcrypt registered [ 8614.004898] Key type .llcrypt registered [ 8614.133162] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 8615.450427] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8615.472362] alg: No test for adler32 (adler32-zlib) [ 8617.430825] Lustre: Lustre: Build Version: 2.17.56_2_gaec046b [ 8618.037222] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 8619.823187] Key type lgssc registered [ 8622.043564] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8637.392168] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8637.447818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8639.502410] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8646.103861] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 8648.685595] LustreError: 203294:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8648.707484] LustreError: 203294:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8653.797184] LustreError: 203293:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8657.294052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8657.965242] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8665.538516] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 8675.631318] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8687.706051] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8702.890406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8703.692949] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8708.996775] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 8712.292405] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 8725.376106] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8736.933514] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 8749.495229] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 8757.043390] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 8763.828163] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 8775.670348] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8775.697220] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8775.702428] Lustre: Skipped 1 previous similar message [ 8775.703269] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8779.674553] Lustre: server umount lustre-OST0000 complete [ 8785.888456] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8785.893258] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8785.903905] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8785.924818] Lustre: Skipped 2 previous similar messages [ 8791.010320] LustreError: 203293:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8791.033493] LustreError: 203293:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8791.151488] Lustre: server umount lustre-MDT0000 complete [ 8796.124686] LustreError: 204030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785405293 with bad export cookie 7391315824738997178 [ 8796.130495] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8796.130916] LustreError: 204538:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8796.135856] LustreError: 204030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8797.335532] Lustre: server umount lustre-MDT0001 complete [ 8818.639296] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 8825.750311] Key type lgssc unregistered [ 8826.310477] LNet: 207863:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8826.329078] LNetError: 207863:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8826.369505] LNet: Removed LNI 192.168.202.113@tcp [ 8827.850777] Key type .llcrypt unregistered [ 8827.856651] Key type ._llcrypt unregistered [ 8864.290524] Key type ._llcrypt registered [ 8864.291789] Key type .llcrypt registered [ 8864.478821] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 8867.134602] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8867.481480] alg: No test for adler32 (adler32-zlib) [ 8869.555145] Lustre: Lustre: Build Version: 2.17.56_2_gaec046b [ 8869.947464] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 8871.815410] Key type lgssc registered [ 8875.989688] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8891.429141] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8891.468739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8893.151520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8898.998231] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 8902.634106] LustreError: 209279:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8902.667575] LustreError: 209279:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8907.747975] LustreError: 209280:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8909.958883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8910.625653] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8917.041283] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 8929.592242] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8939.713717] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8949.691961] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8950.268467] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8958.558385] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 8962.546820] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 8968.537749] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8991.212760] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8991.225641] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8991.278820] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8993.254993] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8993.262282] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8993.338662] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8995.106337] Lustre: server umount lustre-OST0000 complete [ 9002.976034] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9002.982983] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9003.020901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9003.022741] Lustre: Skipped 1 previous similar message [ 9008.608143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9008.613895] Lustre: Skipped 3 previous similar messages [ 9013.734299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9013.738887] Lustre: Skipped 1 previous similar message [ 9016.287983] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9016.534573] Lustre: server umount lustre-MDT0000 complete [ 9017.832657] LustreError: 210723:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9017.871969] LustreError: 210723:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9022.944241] LustreError: 209276:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9022.990632] LustreError: 209276:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9024.418640] LustreError: 209260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785405521 with bad export cookie 1900284396037649750 [ 9024.467348] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9026.347907] Lustre: server umount lustre-MDT0001 complete [ 9044.356757] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 9048.349191] Key type lgssc unregistered [ 9049.053739] LNet: 212970:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9049.064750] LNetError: 212970:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9049.092650] LNet: Removed LNI 192.168.202.113@tcp [ 9051.099463] Key type .llcrypt unregistered [ 9051.103056] Key type ._llcrypt unregistered [ 9081.418615] Key type ._llcrypt registered [ 9081.424193] Key type .llcrypt registered [ 9081.740945] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 9084.033630] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9084.309463] alg: No test for adler32 (adler32-zlib) [ 9086.603507] Lustre: Lustre: Build Version: 2.17.56_2_gaec046b [ 9087.286134] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 9089.279139] Key type lgssc registered [ 9092.921750] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9105.888889] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9105.934802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9108.100150] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9114.855596] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 9117.680624] LustreError: 214252:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9117.743804] LustreError: 214252:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9122.808166] LustreError: 214251:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9127.907337] LustreError: 214252:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9132.836563] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9133.025456] LustreError: 214251:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9133.390232] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9137.902277] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 9148.543654] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9156.411732] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9166.796935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9167.663110] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9176.726411] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 9177.949480] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 9187.402233] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9224.166425] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9224.167376] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9224.175479] LustreError: Skipped 1 previous similar message [ 9224.193213] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9224.205299] Lustre: Skipped 1 previous similar message [ 9226.939885] Lustre: server umount lustre-OST0000 complete [ 9234.400927] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9234.402871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9234.434910] Lustre: Skipped 1 previous similar message [ 9234.443618] Lustre: Skipped 1 previous similar message [ 9239.521190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9239.544968] Lustre: Skipped 2 previous similar messages [ 9243.618326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9243.621743] Lustre: Skipped 1 previous similar message [ 9247.199324] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9247.468414] Lustre: server umount lustre-MDT0000 complete [ 9248.745627] LustreError: 214247:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9248.774301] LustreError: 214247:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9253.352703] LustreError: 214231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785405750 with bad export cookie 6694697094686622695 [ 9253.356176] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9253.378996] LustreError: 214231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9254.084665] Lustre: server umount lustre-MDT0001 complete [ 9270.294280] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 9274.272089] Key type lgssc unregistered [ 9274.935740] LNet: 217975:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9274.944336] LNetError: 217975:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9276.007943] LNet: Removed LNI 192.168.202.113@tcp [ 9278.107405] Key type .llcrypt unregistered [ 9278.111919] Key type ._llcrypt unregistered [ 9311.060378] Key type ._llcrypt registered [ 9311.063751] Key type .llcrypt registered [ 9311.246455] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 9313.145890] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9313.313514] alg: No test for adler32 (adler32-zlib) [ 9314.502574] Lustre: Lustre: Build Version: 2.17.56_2_gaec046b [ 9314.859391] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 9316.567313] Key type lgssc registered [ 9318.475691] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9330.592179] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9330.644584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9332.498555] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9338.152429] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 9341.932655] LustreError: 219246:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9341.957632] LustreError: 219246:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9347.042916] LustreError: 219247:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9351.122487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9351.573031] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9358.244746] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 9369.063592] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9377.074910] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9389.883821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9391.497134] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9401.808650] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 9405.072726] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 9418.977238] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9431.600913] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9431.720532] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9431.949364] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9440.387946] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 9451.071855] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9452.492422] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 9467.182866] Lustre: Permanently deactivating lustre-OST0001 [ 9467.264273] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 9476.331624] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 9485.635878] Lustre: Permanently reactivating lustre-OST0001 [ 9485.664537] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 9485.673264] Lustre: Skipped 2 previous similar messages [ 9494.142334] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9494.163789] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 9494.173371] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 9494.207258] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9494.309694] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 9504.739565] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9504.740040] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9504.749825] LustreError: Skipped 1 previous similar message [ 9504.758468] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9504.786989] Lustre: Skipped 2 previous similar messages [ 9505.777067] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9505.779196] Lustre: Skipped 1 previous similar message [ 9507.314656] Lustre: server umount lustre-OST0001 complete [ 9510.915640] LustreError: 221071:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9510.929071] LustreError: 221071:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9513.443081] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9513.447443] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9513.461491] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9513.463594] Lustre: Skipped 1 previous similar message [ 9514.468680] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9516.003875] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9516.008582] LustreError: 221486:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9516.020733] Lustre: Skipped 2 previous similar messages [ 9516.074144] LustreError: 221486:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9518.996761] Lustre: server umount lustre-OST0000 complete [ 9525.736712] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9525.766154] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9525.813227] Lustre: Skipped 1 previous similar message [ 9525.820734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9530.982330] Lustre: server umount lustre-MDT0000 complete [ 9531.379393] LustreError: 219243:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9531.418450] LustreError: 219243:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9537.994397] LustreError: 223036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785406035 with bad export cookie 16117825200566977123 [ 9538.010592] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9538.022295] LustreError: 223036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9539.499292] Lustre: server umount lustre-MDT0001 complete [ 9559.689261] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 9563.704204] Key type lgssc unregistered [ 9564.180762] LNet: 224343:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9564.189717] LNetError: 224343:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9564.231252] LNet: Removed LNI 192.168.202.113@tcp [ 9565.648526] Key type .llcrypt unregistered [ 9565.650316] Key type ._llcrypt unregistered [ 9620.243659] Key type ._llcrypt registered [ 9620.245407] Key type .llcrypt registered [ 9620.478516] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [ 9621.760213] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9621.824197] alg: No test for adler32 (adler32-zlib) [ 9622.914640] Lustre: Lustre: Build Version: 2.17.56_2_gaec046b [ 9623.280778] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [ 9625.119498] Key type lgssc registered [ 9627.226248] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9642.113424] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9642.163317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9643.890827] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9643.957693] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9644.533664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9653.567133] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 9668.226347] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9668.438167] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9668.519915] Lustre: 226553:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9669.098030] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9676.024563] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 9685.953734] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9693.326705] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9703.228522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9703.510820] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9703.801930] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9713.344193] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 9715.410854] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [ 9728.024627] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9732.444465] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 9736.435643] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 9743.778545] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 9753.588777] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 9764.317321] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 9781.221094] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 9781.241314] Lustre: Skipped 1 previous similar message [ 9821.427458] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 9821.436717] Lustre: Skipped 4 previous similar messages [ 9886.481191] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 9886.496851] Lustre: Skipped 7 previous similar messages [ 9928.673565] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9928.677640] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9928.693873] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9928.710432] LustreError: Skipped 1 previous similar message [ 9928.749406] Lustre: Skipped 1 previous similar message [ 9933.537044] Lustre: server umount lustre-OST0000 complete [ 9941.673884] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9941.686829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9941.707953] Lustre: Skipped 2 previous similar messages [ 9941.724793] Lustre: Skipped 1 previous similar message [ 9947.105896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9947.122261] Lustre: Skipped 1 previous similar message [ 9949.176202] Lustre: server umount lustre-MDT0000 complete [ 9954.789214] LustreError: 226566:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9954.801856] LustreError: 226566:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 9955.456483] LustreError: 226547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785406452 with bad export cookie 7159690397293123113 [ 9955.459928] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9955.468029] LustreError: 226547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9956.510941] Lustre: server umount lustre-MDT0001 complete [ 9978.728781] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [ 9983.546526] Key type lgssc unregistered [ 9984.012583] LNet: 231400:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9984.026435] LNetError: 231400:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9984.046884] LNet: Removed LNI 192.168.202.113@tcp [ 9985.277506] Key type .llcrypt unregistered [ 9985.281926] Key type ._llcrypt unregistered [10015.343926] Key type ._llcrypt registered [10015.356376] Key type .llcrypt registered [10015.613553] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [10018.533546] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10018.669218] alg: No test for adler32 (adler32-zlib) [10020.115376] Lustre: Lustre: Build Version: 2.17.56_2_gaec046b [10020.741143] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [10022.496633] Key type lgssc registered [10024.224711] Lustre: Echo OBD driver; http://www.lustre.org/ [10038.795481] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10038.810316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10040.944151] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10047.541840] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [10051.050937] LustreError: 232817:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10051.079513] LustreError: 232817:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10056.168014] LustreError: 232816:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10059.616583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10060.354840] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10068.544731] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [10086.530750] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10098.228870] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10107.989641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10108.450796] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10110.530228] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [10117.382693] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_default_debug -1 all [10127.380872] Lustre: DEBUG MARKER: oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10142.741213] Lustre: server umount lustre-OST0000 complete [10150.880482] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10150.883820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10150.914365] Lustre: Skipped 1 previous similar message [10150.935981] Lustre: Skipped 1 previous similar message [10156.011092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10156.024196] Lustre: Skipped 1 previous similar message [10159.446991] Lustre: server umount lustre-MDT0000 complete [10161.154201] LustreError: 234051:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10161.184977] LustreError: 234051:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10162.208528] Lustre: 232125:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785406643/real 1785406643] req@000000001ccbeb2e x1872134425060224/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1785406659 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10162.239994] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10165.954837] LustreError: 232796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785406663 with bad export cookie 10749535467921362527 [10165.985269] LustreError: MGC192.168.202.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10167.414957] Lustre: server umount lustre-MDT0001 complete [10189.369122] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [10194.084826] Key type lgssc unregistered [10194.703587] LNet: 236535:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10194.710861] LNetError: 236535:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10194.728066] LNet: Removed LNI 192.168.202.113@tcp [10196.465408] Key type .llcrypt unregistered [10196.473111] Key type ._llcrypt unregistered [10233.620410] Key type ._llcrypt registered [10233.623477] Key type .llcrypt registered [10233.813893] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [10236.425446] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10236.458946] alg: No test for adler32 (adler32-zlib) [10237.854241] Lustre: Lustre: Build Version: 2.17.56_2_gaec046b [10238.444721] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [10240.447224] Key type lgssc registered [10242.292158] Lustre: Echo OBD driver; http://www.lustre.org/ [10516.215922] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [10519.956506] Key type lgssc unregistered [10520.490506] LNet: 238560:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10520.514584] LNetError: 238560:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10520.545603] LNet: Removed LNI 192.168.202.113@tcp [10521.719916] Key type .llcrypt unregistered [10521.727963] Key type ._llcrypt unregistered [10567.582395] Key type ._llcrypt registered [10567.611173] Key type .llcrypt registered [10567.967623] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing set_hostid [10601.787855] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing load_modules_local [10604.976866] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10605.007597] alg: No test for adler32 (adler32-zlib) [10606.443411] Lustre: Lustre: Build Version: 2.17.56_2_gaec046b [10607.019232] LNet: Added LNI 192.168.202.113@tcp [8/256/0/180] [10608.799185] Key type lgssc registered [10610.949710] Lustre: Echo OBD driver; http://www.lustre.org/ [10652.557733] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 10213 sec ========= 06:25:47 (1785407147) [10655.290538] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 06:25:50 (1785407150) === [10659.436553] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 06:25:54 (1785407154) === [10721.965788] Lustre: DEBUG MARKER: oleg213-server.virtnet: executing unload_modules_local [10726.200457] Key type lgssc unregistered [10726.571583] LNet: 243811:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10726.581138] LNetError: 243811:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10726.623086] LNet: Removed LNI 192.168.202.113@tcp [10727.820603] Key type .llcrypt unregistered [10727.824715] Key type ._llcrypt unregistered