[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 559036649 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002301] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004012] kvm-guest: setup PV IPIs [ 0.006943] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009006] pid_max: default: 32768 minimum: 301 [ 0.010115] LSM: Security Framework initializing [ 0.011051] Yama: becoming mindful. [ 0.012028] SELinux: Initializing. [ 0.014045] *** VALIDATE selinux *** [ 0.020584] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024550] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025150] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026101] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027101] *** VALIDATE tmpfs *** [ 0.028401] *** VALIDATE proc *** [ 0.029193] *** VALIDATE cgroup *** [ 0.030005] *** VALIDATE cgroup2 *** [ 0.031187] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.032107] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.033009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.034028] Spectre V2 : User space: Vulnerable [ 0.035006] Speculative Store Bypass: Vulnerable [ 0.038160] debug: unmapping init [mem 0xffffffff99259000-0xffffffff99260fff] [ 0.040667] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.041675] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.042023] ... version: 2 [ 0.043012] ... bit width: 48 [ 0.044011] ... generic registers: 4 [ 0.045008] ... value mask: 0000ffffffffffff [ 0.046009] ... max period: 00007fffffffffff [ 0.047013] ... fixed-purpose events: 3 [ 0.048012] ... event mask: 000000070000000f [ 0.049288] rcu: Hierarchical SRCU implementation. [ 0.051338] smp: Bringing up secondary CPUs ... [ 0.052542] x86: Booting SMP configuration: [ 0.053024] .... node #0, CPUs: #1 #2 #3 [ 0.056329] smp: Brought up 1 node, 4 CPUs [ 0.058011] smpboot: Max logical packages: 1 [ 0.059086] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.137429] node 0 deferred pages initialised in 76ms [ 0.142169] devtmpfs: initialized [ 0.143200] x86/mm: Memory block size: 128MB [ 0.146000] gcov: version magic: 0x41383552 [ 0.150340] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.151092] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.152276] pinctrl core: initialized pinctrl subsystem [ 0.153243] [ 0.154000] ************************************************************* [ 0.155013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.156021] ** ** [ 0.157016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.158013] ** ** [ 0.159011] ** This means that this kernel is built to expose internal ** [ 0.160011] ** IOMMU data structures, which may compromise security on ** [ 0.161013] ** your system. ** [ 0.162010] ** ** [ 0.163011] ** If you see this message and you are not debugging the ** [ 0.164010] ** kernel, report this immediately to your vendor! ** [ 0.165014] ** ** [ 0.166011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.167015] ************************************************************* [ 0.168757] NET: Registered protocol family 16 [ 0.170065] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.173210] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.178062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.182161] cpuidle: using governor menu [ 0.184864] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.187794] PCI: Using configuration type 1 for base access [ 0.191129] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.204059] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.206016] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.209122] cryptd: max_cpu_qlen set to 1000 [ 0.213263] ACPI: Added _OSI(Module Device) [ 0.215009] ACPI: Added _OSI(Processor Device) [ 0.217027] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.219011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.224995] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.232300] ACPI: Interpreter enabled [ 0.233096] ACPI: PM: (supports S0 S3 S4 S5) [ 0.235011] ACPI: Using IOAPIC for interrupt routing [ 0.237084] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.241372] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.252451] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.254034] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.257016] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.260144] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.265552] acpiphp: Slot [2] registered [ 0.267168] acpiphp: Slot [5] registered [ 0.269166] acpiphp: Slot [6] registered [ 0.273148] acpiphp: Slot [7] registered [ 0.274092] acpiphp: Slot [8] registered [ 0.276086] acpiphp: Slot [9] registered [ 0.277091] acpiphp: Slot [10] registered [ 0.278096] acpiphp: Slot [3] registered [ 0.282136] acpiphp: Slot [4] registered [ 0.284117] acpiphp: Slot [11] registered [ 0.285087] acpiphp: Slot [12] registered [ 0.286081] acpiphp: Slot [13] registered [ 0.288228] acpiphp: Slot [14] registered [ 0.290255] acpiphp: Slot [15] registered [ 0.292090] acpiphp: Slot [16] registered [ 0.294127] acpiphp: Slot [17] registered [ 0.296077] acpiphp: Slot [18] registered [ 0.298090] acpiphp: Slot [19] registered [ 0.299196] acpiphp: Slot [20] registered [ 0.301126] acpiphp: Slot [21] registered [ 0.303094] acpiphp: Slot [22] registered [ 0.305087] acpiphp: Slot [23] registered [ 0.306111] acpiphp: Slot [24] registered [ 0.308090] acpiphp: Slot [25] registered [ 0.310116] acpiphp: Slot [26] registered [ 0.312085] acpiphp: Slot [27] registered [ 0.313090] acpiphp: Slot [28] registered [ 0.315082] acpiphp: Slot [29] registered [ 0.317131] acpiphp: Slot [30] registered [ 0.319106] acpiphp: Slot [31] registered [ 0.321085] PCI host bridge to bus 0000:00 [ 0.324018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.327023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.331022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.335032] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.339030] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.344080] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.346191] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.350176] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.355336] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.366747] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.373012] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.376017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.379017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.382017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.386227] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.390131] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.394045] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.397912] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.404014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.422021] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.428021] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.435206] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.443015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.450015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.471015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.487699] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.495014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.505020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.528026] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.537000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.552019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.569016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.615016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.639390] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.648014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.667019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.716018] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.738885] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.753022] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.762021] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.784023] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.796701] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.817017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.834018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.873023] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.886703] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.888363] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.891457] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.894443] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.898371] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.903218] iommu: Default domain type: Passthrough [ 0.905529] SCSI subsystem initialized [ 0.907184] ACPI: bus type USB registered [ 0.909184] usbcore: registered new interface driver usbfs [ 0.911115] usbcore: registered new interface driver hub [ 0.913304] usbcore: registered new device driver usb [ 0.916228] pps_core: LinuxPPS API ver. 1 registered [ 0.918012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.921133] PTP clock support registered [ 0.924159] EDAC MC: Ver: 3.0.0 [ 0.927187] PCI: Using ACPI for IRQ routing [ 0.929977] NetLabel: Initializing [ 0.931022] NetLabel: domain hash size = 128 [ 0.933016] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.936152] NetLabel: unlabeled traffic allowed by default [ 0.939053] vgaarb: loaded [ 0.943333] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.945020] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.954212] clocksource: Switched to clocksource kvm-clock [ 1.069585] VFS: Disk quotas dquot_6.6.0 [ 1.072036] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.076322] *** VALIDATE ramfs *** [ 1.078432] *** VALIDATE hugetlbfs *** [ 1.081249] pnp: PnP ACPI init [ 1.084995] pnp: PnP ACPI: found 6 devices [ 1.104823] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.108354] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.110720] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.112587] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.114997] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.117932] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.120160] NET: Registered protocol family 2 [ 1.122412] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.128252] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.134617] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.139581] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.143101] TCP: Hash tables configured (established 65536 bind 65536) [ 1.146317] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.151099] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.153904] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.157488] NET: Registered protocol family 1 [ 1.160326] RPC: Registered named UNIX socket transport module. [ 1.162915] RPC: Registered udp transport module. [ 1.164677] RPC: Registered tcp transport module. [ 1.166969] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.169419] NET: Registered protocol family 44 [ 1.170782] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.172855] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.175372] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.177863] PCI: CLS 0 bytes, default 64 [ 1.180364] Unpacking initramfs... [ 2.704453] debug: unmapping init [mem 0xffff8d453cc54000-0xffff8d453ffbffff] [ 2.713029] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.715847] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.720907] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.246918] Initialise system trusted keyrings [ 3.249101] Key type blacklist registered [ 3.251916] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.264712] zbud: loaded [ 3.270724] *** VALIDATE nfs *** [ 3.272616] *** VALIDATE nfs4 *** [ 3.275209] pstore: using deflate compression [ 3.279574] Platform Keyring initialized [ 3.403328] NET: Registered protocol family 38 [ 3.405154] Key type asymmetric registered [ 3.406673] Asymmetric key parser 'x509' registered [ 3.408588] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.412331] io scheduler mq-deadline registered [ 3.414159] io scheduler kyber registered [ 3.415721] io scheduler bfq registered [ 3.417894] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.422153] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.426193] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.429924] ACPI: Power Button [PWRF] [ 3.438521] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.446835] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.469416] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.479674] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.509610] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.542074] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.571169] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.577558] Non-volatile memory driver v1.3 [ 3.579593] Linux agpgart interface v0.103 [ 3.621252] virtio_blk virtio1: [vda] 149952 512-byte logical blocks (76.8 MB/73.2 MiB) [ 3.624784] vda: detected capacity change from 0 to 76775424 [ 3.650865] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.657352] vdb: detected capacity change from 0 to 1073741824 [ 3.680940] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.684952] vdc: detected capacity change from 0 to 2621440000 [ 3.710144] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.713680] vdd: detected capacity change from 0 to 2621440000 [ 3.735153] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.739756] vde: detected capacity change from 0 to 4294967296 [ 3.758186] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.761625] vdf: detected capacity change from 0 to 4294967296 [ 3.774453] libphy: Fixed MDIO Bus: probed [ 3.780881] usbcore: registered new interface driver usbserial_generic [ 3.784490] usbserial: USB Serial support registered for generic [ 3.787206] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.792289] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.795144] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.798537] mousedev: PS/2 mouse device common for all mice [ 3.803358] rtc_cmos 00:05: RTC can wake from S4 [ 3.807496] rtc_cmos 00:05: registered as rtc0 [ 3.809648] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.812973] intel_pstate: CPU model not supported [ 3.815490] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.817041] hid: raw HID events driver (C) Jiri Kosina [ 3.823677] usbcore: registered new interface driver usbhid [ 3.825473] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.826428] usbhid: USB HID core driver [ 3.826599] drop_monitor: Initializing network drop monitor service [ 3.831391] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.833503] Initializing XFRM netlink socket [ 3.838674] NET: Registered protocol family 10 [ 3.841513] Segment Routing with IPv6 [ 3.843595] NET: Registered protocol family 17 [ 3.846104] mpls_gso: MPLS GSO support [ 3.851375] RAS: Correctable Errors collector initialized. [ 3.853753] AVX version of gcm_enc/dec engaged. [ 3.856221] AES CTR mode by8 optimization enabled [ 3.945910] sched_clock: Marking stable (3945830112, 0)->(4898399746, -952569634) [ 3.950950] registered taskstats version 1 [ 3.952957] Loading compiled-in X.509 certificates [ 3.954835] zswap: loaded using pool lzo/zbud [ 3.981098] Key type big_key registered [ 3.994975] Key type encrypted registered [ 3.998324] ima: No TPM chip found, activating TPM-bypass! [ 4.001530] ima: Allocated hash algorithm: sha1 [ 4.003626] ima: No architecture policies found [ 4.005494] evm: Initialising EVM extended attributes: [ 4.007858] evm: security.selinux [ 4.009637] evm: security.ima [ 4.011307] evm: security.capability [ 4.012956] evm: HMAC attrs: 0x1 [ 4.016451] rtc_cmos 00:05: setting system clock to 2026-09-15 13:41:53 UTC (1789479713) [ 4.024438] debug: unmapping init [mem 0xffffffff9a203000-0xffffffff9a3fffff] [ 4.028237] debug: unmapping init [mem 0xffffffff98f82000-0xffffffff99258fff] [ 4.038604] Write protecting the kernel read-only data: 28672k [ 4.043191] debug: unmapping init [mem 0xffffffff97603000-0xffffffff977fffff] [ 4.047388] debug: unmapping init [mem 0xffffffff97f14000-0xffffffff97ffffff] [ 4.091574] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.101705] systemd[1]: Detected virtualization kvm. [ 4.104349] systemd[1]: Detected architecture x86-64. [ 4.106782] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.135833] systemd[1]: No hostname configured. [ 4.137672] systemd[1]: Set hostname to . [ 4.140596] random: systemd: uninitialized urandom read (16 bytes read) [ 4.143661] systemd[1]: Initializing machine ID from random generator. [ 4.186214] random: ln: uninitialized urandom read (6 bytes read) [ 4.290882] random: systemd: uninitialized urandom read (16 bytes read) [ 4.302174] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 4.315163] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 4.322423] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Journal Service... Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.073470] device-mapper: uevent: version 1.0.3 [ 5.076406] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK [0[ 5.853646] random: fast init done m] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.903070] virtio_net virtio0 ens2: renamed from eth0 [ 5.944472] scsi host0: ata_piix [ 5.971862] scsi host1: ata_piix [ 5.974640] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.978845] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.520736] random: crng init done [ 10.522497] random: 7 urandom warning(s) missed due to ratelimiting [ 10.565891] dracut-initqueue[589]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 11.283850] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.749191] printk: systemd: 26 output lines suppressed due to ratelimiting [ 13.068869] SELinux: Disabled at runtime. [ 13.136182] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 13.147953] systemd[1]: Detected virtualization kvm. [ 13.149968] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.776431] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.781441] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.790305] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.795764] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.801440] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.812696] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.821384] systemd[1]: Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Remount Root and Kernel File Systems... Activating swap /dev/disk/by-label/SWAP... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-getty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Reached target rpc_pipefs.target. [ 13.911083] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Initrd File Systems. Starting Apply Kernel Variables... Mounting POSIX Message Queue File System... Mounting Kernel Debug File System... Mounting Huge Pages File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 14.442196] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 14.907718] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.936972] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 15.121818] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 15.137185] EDAC sbridge: Ver: 1.1.2 [ 16.966783] Key type dns_resolver registered [ 17.303902] NFS: Registering the id_resolver key type [ 17.306891] Key type id_resolver registered [ 17.309335] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg631-server login: [ 50.755106] libcfs: loading out-of-tree module taints kernel. [ 50.794563] Key type ._llcrypt registered [ 50.797261] Key type .llcrypt registered [ 50.884609] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_hostid [ 70.930437] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 71.016072] hrtimer: interrupt took 5060561 ns [ 72.777089] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 72.785751] alg: No test for adler32 (adler32-zlib) [ 74.555872] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [ 75.819770] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 77.713966] Key type lgssc registered [ 80.257383] Lustre: Echo OBD driver; http://www.lustre.org/ [ 100.072519] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 148.830418] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 162.767989] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 162.798489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 164.011447] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 164.042315] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 164.121193] Lustre: lustre-MDT0000: new disk, initializing [ 164.212964] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 164.242297] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 168.377214] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 183.883026] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 184.095262] Lustre: 6526:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 184.159211] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 184.165610] Lustre: Skipped 1 previous similar message [ 184.364246] Lustre: lustre-MDT0001: new disk, initializing [ 184.576100] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 184.626910] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 184.665079] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 190.334186] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 195.997989] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 207.947975] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 208.178647] Lustre: lustre-OST0000: new disk, initializing [ 208.182682] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 208.188306] Lustre: 8463:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 208.282727] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 214.705353] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 217.645090] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 217.671942] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 217.767261] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 232.950536] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 233.149971] Lustre: lustre-OST0001: new disk, initializing [ 233.156077] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 233.163441] Lustre: 9537:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 233.266046] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 241.256269] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 241.270893] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 241.400056] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 243.452970] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 258.021169] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 266.986956] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 273.516115] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing check_logdir /tmp/testlogs/ [ 279.018900] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing yml_node [ 284.354358] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 287.137461] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 289.683725] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 291.511779] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Sep 15 09:46:39 EDT 2026 [ 310.841529] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 312.771807] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 333.281866] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 333.303544] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 333.315561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 333.805892] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 333.825802] Lustre: Skipped 2 previous similar messages [ 333.831339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 333.833915] Lustre: Skipped 2 previous similar messages [ 335.208565] Lustre: server umount lustre-MDT0000 complete [ 338.679073] LustreError: 10352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789480048 with bad export cookie 15129102151557452727 [ 338.683367] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 338.690649] LustreError: 10352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 338.912798] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 338.916316] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 338.930127] Lustre: Skipped 1 previous similar message [ 344.032166] LustreError: 6532:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 344.033406] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 344.070762] LustreError: 6532:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 344.085047] Lustre: Skipped 2 previous similar messages [ 345.255635] Lustre: server umount lustre-MDT0001 complete [ 349.184991] Lustre: server umount lustre-OST0000 complete [ 352.986352] Lustre: server umount lustre-OST0001 complete [ 359.564463] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_hostid [ 370.342175] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 406.415065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 406.687924] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 406.729886] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 406.827655] Lustre: lustre-MDT0000: new disk, initializing [ 406.929303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 406.949272] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 411.582187] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 421.875395] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 422.024731] Lustre: 15090:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 422.086929] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 422.090766] Lustre: Skipped 1 previous similar message [ 422.169914] Lustre: lustre-MDT0001: new disk, initializing [ 422.280514] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 422.326653] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 422.339809] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 426.260382] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 435.739671] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 441.231557] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 447.469072] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 447.740904] Lustre: lustre-OST0000: new disk, initializing [ 447.747262] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 447.764194] Lustre: 17060:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 447.860347] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 449.511089] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 449.528206] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 449.650442] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 453.536641] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 462.523247] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 466.800662] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 467.005391] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 471.906628] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 472.150135] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 475.107349] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 475.117316] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 475.156127] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 480.088440] Lustre: server umount lustre-OST0000 complete [ 485.344893] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 485.346914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 485.366256] Lustre: Skipped 2 previous similar messages [ 485.376735] Lustre: Skipped 2 previous similar messages [ 490.248991] Lustre: server umount lustre-MDT0000 complete [ 490.467531] LustreError: 15970:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 490.493644] LustreError: 15970:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 493.729836] LustreError: 15083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789480203 with bad export cookie 15129102151557454260 [ 493.741672] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 493.745255] LustreError: 15083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 494.345283] Lustre: server umount lustre-MDT0001 complete [ 497.607668] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 09:50:05 (1789480205) [ 503.663607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 504.189757] LustreError: 19325:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 504.243286] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 508.276181] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 509.418604] LustreError: 19326:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 514.542834] LustreError: 19325:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 517.723701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 522.766889] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 530.864254] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 537.800432] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 544.190457] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 550.392468] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 558.340924] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 569.836879] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 569.847793] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 569.871359] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 573.430357] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 583.135390] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 583.567643] Lustre: server umount lustre-OST0000 complete [ 588.769366] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 588.789842] Lustre: Skipped 2 previous similar messages [ 593.897656] LustreError: 20540:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 594.046580] Lustre: server umount lustre-MDT0000 complete [ 598.404128] LustreError: 19307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789480307 with bad export cookie 15129102151557455142 [ 598.430387] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 599.084422] Lustre: server umount lustre-MDT0001 complete [ 611.107201] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 614.257917] Key type lgssc unregistered [ 614.618644] LNet: 22857:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 614.626724] LNetError: 22857:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 614.647856] LNet: Removed LNI 192.168.206.131@tcp [ 615.904135] Key type .llcrypt unregistered [ 615.910325] Key type ._llcrypt unregistered [ 637.284358] Key type ._llcrypt registered [ 637.286065] Key type .llcrypt registered [ 637.405528] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 638.203029] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 638.381887] alg: No test for adler32 (adler32-zlib) [ 639.392434] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [ 639.646692] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 641.295197] Key type lgssc registered [ 642.510903] Lustre: Echo OBD driver; http://www.lustre.org/ [ 650.349940] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 650.364552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 651.789569] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 656.176636] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 661.993296] LustreError: 24267:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 662.020344] LustreError: 24267:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 664.037732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 664.305397] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 668.567174] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 676.132516] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 681.652842] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 687.495626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 687.814525] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 694.830581] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 697.066053] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 701.814064] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 717.594955] Lustre: server umount lustre-OST0000 complete [ 722.917700] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 722.920570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 722.935417] Lustre: Skipped 1 previous similar message [ 728.036233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 728.039506] Lustre: Skipped 2 previous similar messages [ 731.221721] Lustre: server umount lustre-MDT0000 complete [ 733.153891] LustreError: 24266:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 733.183892] LustreError: 24266:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 734.179829] Lustre: 23578:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789480427/real 1789480427] req@000000005d147e44 x1876406146462336/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789480443 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 734.219262] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 734.244448] LustreError: 25509:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 734.277481] LustreError: 25509:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 734.936763] LustreError: 24245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789480444 with bad export cookie 11041353184663398660 [ 734.942646] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 734.951204] LustreError: 24245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 735.269338] Lustre: server umount lustre-MDT0001 complete [ 747.625730] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 750.545016] Key type lgssc unregistered [ 750.823887] LNet: 28242:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 750.838735] LNetError: 28242:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 750.868454] LNet: Removed LNI 192.168.206.131@tcp [ 751.894454] Key type .llcrypt unregistered [ 751.899886] Key type ._llcrypt unregistered [ 775.492264] Key type ._llcrypt registered [ 775.494514] Key type .llcrypt registered [ 775.636184] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 776.812194] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 776.907297] alg: No test for adler32 (adler32-zlib) [ 778.054098] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [ 778.247289] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 779.895789] Key type lgssc registered [ 780.791672] Lustre: Echo OBD driver; http://www.lustre.org/ [ 787.573604] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 787.596622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 788.869726] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 793.129546] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 799.205764] LustreError: 29653:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 799.220573] LustreError: 29653:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 801.669969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 802.121921] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 806.753071] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 814.298126] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 820.201265] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 834.100341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 834.548991] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 840.603329] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 848.568711] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 859.126636] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 864.225740] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 864.229871] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 864.232808] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 864.259962] Lustre: Skipped 1 previous similar message [ 865.735734] Lustre: server umount lustre-OST0000 complete [ 868.837816] LustreError: 28963:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000e320676f x1876406291169536/t0(0) o41->lustre-MDT0001-osp-MDT0000@0@lo:24/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 868.839917] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 868.864785] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 868.871270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 868.877778] Lustre: Skipped 1 previous similar message [ 874.465609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 874.478355] Lustre: Skipped 3 previous similar messages [ 879.588424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 879.600302] Lustre: Skipped 1 previous similar message [ 883.168473] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 883.456781] Lustre: server umount lustre-MDT0000 complete [ 883.681578] LustreError: 30380:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 883.693273] LustreError: 30380:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 886.861239] LustreError: 29634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789480596 with bad export cookie 5410159908807078908 [ 886.866132] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 886.872557] LustreError: 29634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 887.303085] Lustre: server umount lustre-MDT0001 complete [ 896.509878] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 899.128334] Key type lgssc unregistered [ 899.440719] LNet: 33548:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 899.448958] LNetError: 33548:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 899.472752] LNet: Removed LNI 192.168.206.131@tcp [ 900.281222] Key type .llcrypt unregistered [ 900.287830] Key type ._llcrypt unregistered [ 920.060966] Key type ._llcrypt registered [ 920.067211] Key type .llcrypt registered [ 920.185777] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 921.090665] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 921.209859] alg: No test for adler32 (adler32-zlib) [ 922.244733] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [ 922.504730] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 924.192177] Key type lgssc registered [ 925.491789] Lustre: Echo OBD driver; http://www.lustre.org/ [ 932.948930] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 932.969534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 934.477056] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 938.562480] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 943.586893] LustreError: 34937:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 943.614088] LustreError: 34937:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 946.709559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 947.115896] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 951.463259] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 958.181231] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 963.467090] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 969.371896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 969.670473] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 975.242906] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 978.936586] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 981.955192] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 991.717936] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 991.726781] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 991.739802] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 992.224983] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 992.235328] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 992.247256] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 994.276046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 994.282423] Lustre: Skipped 1 previous similar message [ 995.860414] Lustre: server umount lustre-OST0000 complete [ 999.402639] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 999.408574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 999.420588] Lustre: Skipped 1 previous similar message [ 1004.512726] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1004.519270] Lustre: Skipped 2 previous similar messages [ 1013.215110] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1013.389717] Lustre: server umount lustre-MDT0000 complete [ 1014.751825] LustreError: 36373:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1014.764267] LustreError: 36373:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1016.453363] LustreError: 34920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789480725 with bad export cookie 3283736007760206711 [ 1016.455640] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1016.470132] LustreError: 34920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1016.880352] Lustre: server umount lustre-MDT0001 complete [ 1026.270947] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 1028.544506] Key type lgssc unregistered [ 1028.782528] LNet: 38471:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1028.787401] LNetError: 38471:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1028.811531] LNet: Removed LNI 192.168.206.131@tcp [ 1029.511926] Key type .llcrypt unregistered [ 1029.517502] Key type ._llcrypt unregistered [ 1048.018549] Key type ._llcrypt registered [ 1048.020756] Key type .llcrypt registered [ 1048.101596] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 1048.966856] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1048.984102] alg: No test for adler32 (adler32-zlib) [ 1050.023450] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [ 1050.229073] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 1051.879268] Key type lgssc registered [ 1052.759387] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1058.482932] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1058.508974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1059.795211] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1062.895897] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1069.409230] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1069.658280] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1073.003155] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1078.700985] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1082.931095] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1087.956991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1088.219681] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1092.822119] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1097.884128] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1105.654261] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1113.682420] Lustre: lustre-OST0000: Not available for connect from 192.168.206.31@tcp (stopping) [ 1116.131931] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1116.136394] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1116.153265] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1116.156871] LustreError: Skipped 1 previous similar message [ 1118.593537] Lustre: server umount lustre-OST0000 complete [ 1136.098866] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1136.107304] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1136.120558] Lustre: Skipped 1 previous similar message [ 1136.127991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1136.136954] Lustre: Skipped 1 previous similar message [ 1141.631458] Lustre: server umount lustre-MDT0000 complete [ 1142.758229] LustreError: 39862:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1142.767419] LustreError: 39862:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1144.056835] LustreError: 42835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789480853 with bad export cookie 11313107145002090224 [ 1144.059246] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1144.064166] LustreError: 42835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1150.320494] Lustre: server umount lustre-MDT0001 complete [ 1158.545715] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 1160.695282] Key type lgssc unregistered [ 1160.961124] LNet: 43600:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1160.968488] LNetError: 43600:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1162.023067] LNet: Removed LNI 192.168.206.131@tcp [ 1162.688551] Key type .llcrypt unregistered [ 1162.690537] Key type ._llcrypt unregistered [ 1179.106398] Key type ._llcrypt registered [ 1179.108392] Key type .llcrypt registered [ 1179.191110] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 1179.969893] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1180.030065] alg: No test for adler32 (adler32-zlib) [ 1181.023537] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [ 1181.193450] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 1182.863190] Key type lgssc registered [ 1183.682610] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1189.027369] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1189.040535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1190.348420] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1193.448775] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1199.388071] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1199.620311] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1202.863156] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1208.328477] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1212.643314] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1217.821177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1218.072693] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1223.157479] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1228.840582] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1236.528476] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1244.735781] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.31@tcp (stopping) [ 1245.505619] Lustre: server umount lustre-MDT0000 complete [ 1245.673632] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1245.678292] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1245.686430] LustreError: 44984:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1245.694760] LustreError: 44984:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1246.697325] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1246.698975] LustreError: 44985:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1246.711623] Lustre: Skipped 1 previous similar message [ 1246.722280] LustreError: 44985:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1248.651411] LustreError: 44966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789480958 with bad export cookie 5368797550084979570 [ 1248.661267] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1248.662270] LustreError: 44966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1249.843143] LustreError: 44981:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1249.859422] LustreError: 44981:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1249.876666] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.31@tcp (stopping) [ 1251.817230] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1251.839527] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1253.857321] LustreError: 44981:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1253.859720] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1253.881259] LustreError: 44981:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1255.041884] Lustre: server umount lustre-MDT0001 complete [ 1275.082212] Lustre: server umount lustre-OST0000 complete [ 1287.542858] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 1289.478522] Key type lgssc unregistered [ 1289.712128] LNet: 48861:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1289.717047] LNetError: 48861:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1289.729589] LNet: Removed LNI 192.168.206.131@tcp [ 1290.303785] Key type .llcrypt unregistered [ 1290.308936] Key type ._llcrypt unregistered [ 1308.040527] Key type ._llcrypt registered [ 1308.042368] Key type .llcrypt registered [ 1308.126853] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 1308.879971] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1308.898237] alg: No test for adler32 (adler32-zlib) [ 1309.873830] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [ 1310.054960] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 1311.679217] Key type lgssc registered [ 1312.474111] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1318.014715] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1318.031720] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1333.537097] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1338.655342] LustreError: 50224:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.131@tcp: failed processing log, type 4: rc = -110 [ 1365.279202] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1369.964298] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1375.579720] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1542.330922] Lustre: server umount lustre-OST0000 complete [ 1553.808659] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 1556.014801] Key type lgssc unregistered [ 1556.247692] LNet: 51898:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1556.253373] LNetError: 51898:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1556.269095] LNet: Removed LNI 192.168.206.131@tcp [ 1556.843626] Key type .llcrypt unregistered [ 1556.846451] Key type ._llcrypt unregistered [ 1573.114599] Key type ._llcrypt registered [ 1573.117739] Key type .llcrypt registered [ 1573.211973] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 1573.810061] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1573.937413] alg: No test for adler32 (adler32-zlib) [ 1574.919621] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [ 1575.102443] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 1576.751975] Key type lgssc registered [ 1577.519420] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1582.515427] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1582.526924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1583.803181] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1586.327696] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1591.819416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1592.054459] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1594.993706] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1600.213584] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1604.312934] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1609.020685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1609.305809] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1610.355082] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1613.778817] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1618.996135] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1620.962414] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1620.969079] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1620.986260] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1623.012821] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1623.013563] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1623.034568] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1623.037513] Lustre: Skipped 1 previous similar message [ 1626.712929] Lustre: server umount lustre-OST0000 complete [ 1633.254493] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1633.263072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1633.263171] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1633.267330] Lustre: Skipped 1 previous similar message [ 1633.287018] Lustre: Skipped 1 previous similar message [ 1635.465365] Lustre: server umount lustre-MDT0000 complete [ 1638.314227] LustreError: 54018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789481347 with bad export cookie 9310230590285859019 [ 1638.321464] LustreError: 54018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1638.321597] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1638.560900] Lustre: server umount lustre-MDT0001 complete [ 1646.566516] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 1648.618749] Key type lgssc unregistered [ 1648.829854] LNet: 56811:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1648.837105] LNetError: 56811:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1648.853864] LNet: Removed LNI 192.168.206.131@tcp [ 1649.543580] Key type .llcrypt unregistered [ 1649.550838] Key type ._llcrypt unregistered [ 1666.854801] Key type ._llcrypt registered [ 1666.856980] Key type .llcrypt registered [ 1666.958853] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 1667.756598] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1667.847208] alg: No test for adler32 (adler32-zlib) [ 1668.897332] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [ 1669.116604] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 1670.783198] Key type lgssc registered [ 1671.624764] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1677.300839] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1677.315585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1692.831798] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1697.951356] LustreError: 58173:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.131@tcp: failed processing log, type 4: rc = -110 [ 1724.578193] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1728.631860] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1733.209305] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1737.637401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1737.987130] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1738.050808] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1741.358410] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1744.358270] LustreError: 59210:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1744.377103] LustreError: 59210:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 1746.881380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1747.133738] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1750.002797] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1754.735546] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1758.333957] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1762.786542] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1762.789373] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1762.803648] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1762.808676] Lustre: Skipped 1 previous similar message [ 1765.752712] Lustre: server umount lustre-OST0000 complete [ 1783.267897] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1783.268772] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1783.284339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1783.292641] Lustre: Skipped 2 previous similar messages [ 1784.353430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1784.359721] Lustre: Skipped 2 previous similar messages [ 1789.185819] Lustre: server umount lustre-MDT0000 complete [ 1789.411825] LustreError: 59214:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1789.427076] LustreError: 59214:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1791.382775] LustreError: 58180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789481500 with bad export cookie 5639054934306834749 [ 1791.388489] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1791.396897] LustreError: 58180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1797.725361] Lustre: server umount lustre-MDT0001 complete [ 1804.508608] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 1806.328971] Key type lgssc unregistered [ 1806.547739] LNet: 61914:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1806.551950] LNetError: 61914:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1806.562429] LNet: Removed LNI 192.168.206.131@tcp [ 1807.198608] Key type .llcrypt unregistered [ 1807.201947] Key type ._llcrypt unregistered [ 1822.226764] Key type ._llcrypt registered [ 1822.228675] Key type .llcrypt registered [ 1822.314442] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 1822.901313] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1822.938984] alg: No test for adler32 (adler32-zlib) [ 1823.932077] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [ 1824.122994] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 1825.775406] Key type lgssc registered [ 1826.525501] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1831.620525] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1831.634867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1832.906065] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1835.293459] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1840.652464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1840.886482] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1844.022736] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1848.704148] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1852.283934] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1856.753206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1856.967189] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1858.043021] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1860.559742] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1865.034968] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1881.571720] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1881.572068] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1881.575399] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1881.591253] Lustre: Skipped 1 previous similar message [ 1886.084750] Lustre: server umount lustre-OST0000 complete [ 1891.809144] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1891.818570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1891.822075] Lustre: Skipped 1 previous similar message [ 1891.835035] Lustre: Skipped 2 previous similar messages [ 1894.497783] Lustre: server umount lustre-MDT0000 complete [ 1896.928445] LustreError: 65084:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1896.952633] LustreError: 65084:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1896.979624] LustreError: 63280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789481606 with bad export cookie 6187885761346621932 [ 1896.981980] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1896.992094] LustreError: 63280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1897.003995] LustreError: 62613:0:(import.c:762:ptlrpc_connect_import_locked()) already connecting [ 1897.205564] Lustre: server umount lustre-MDT0001 complete [ 1903.650312] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 1905.417905] Key type lgssc unregistered [ 1905.643645] LNet: 66851:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1905.650114] LNetError: 66851:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1905.663967] LNet: Removed LNI 192.168.206.131@tcp [ 1906.201361] Key type .llcrypt unregistered [ 1906.203276] Key type ._llcrypt unregistered [ 1921.663466] Key type ._llcrypt registered [ 1921.664994] Key type .llcrypt registered [ 1921.745152] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 1922.378616] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1922.398951] alg: No test for adler32 (adler32-zlib) [ 1923.354831] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [ 1923.543897] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 1925.175570] Key type lgssc registered [ 1925.882651] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1931.660909] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 10:14:00 (1789481640) [ 1936.128394] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1936.139673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1937.404749] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1940.174252] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1945.941726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1946.203402] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1949.034963] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1954.278466] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1957.746942] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1962.225158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1962.427889] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1963.436299] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1966.839080] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1972.020486] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1977.158187] Lustre: Failing over lustre-MDT0000 [ 1977.386957] Lustre: server umount lustre-MDT0000 complete [ 1979.461277] LustreError: 68525:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1979.472156] LustreError: 68525:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1980.907568] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1980.909097] LustreError: 69256:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1980.919302] Lustre: Skipped 1 previous similar message [ 1983.215476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1983.313855] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1983.408354] Lustre: *** cfs_fail_loc=135, val=0*** [ 1983.471447] LustreError: 71614:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 1983.476240] LustreError: 71614:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.131@tcp: cfg command failed: rc = -2 [ 1983.482183] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1983.482183] [ 1983.488734] LustreError: MGC192.168.206.131@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1983.497450] LustreError: 71602:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1983.502562] LustreError: 71602:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 1983.509090] LustreError: 71602:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 1983.513719] LustreError: 71602:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 1983.533129] Lustre: server umount lustre-MDT0000 complete [ 1983.537832] LustreError: 71602:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1984.556570] LustreError: 68526:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1984.569097] LustreError: 68526:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1987.801416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1987.899493] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1988.061797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1988.102821] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1989.676451] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1990.957277] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 1993.202065] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1993.216709] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 1993.263172] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 1998.310676] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1998.312722] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1998.322592] Lustre: Skipped 2 previous similar messages [ 2002.514703] Lustre: server umount lustre-OST0000 complete [ 2008.545750] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2008.546597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2008.546671] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2008.556353] Lustre: Skipped 1 previous similar message [ 2008.568828] Lustre: Skipped 2 previous similar messages [ 2013.664452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2017.764818] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2017.769363] Lustre: Skipped 2 previous similar messages [ 2019.295250] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2019.396422] Lustre: server umount lustre-MDT0000 complete [ 2021.573598] LustreError: 69263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789481731 with bad export cookie 10117942793554873200 [ 2021.576637] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2021.583070] LustreError: 69263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2021.805509] Lustre: server umount lustre-MDT0001 complete [ 2028.387945] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 2030.129487] Key type lgssc unregistered [ 2030.336634] LNet: 73390:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2030.345175] LNetError: 73390:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2030.364422] LNet: Removed LNI 192.168.206.131@tcp [ 2030.912333] Key type .llcrypt unregistered [ 2030.914408] Key type ._llcrypt unregistered [ 2044.886541] Key type ._llcrypt registered [ 2044.890448] Key type .llcrypt registered [ 2044.958180] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 2045.396767] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2045.406698] alg: No test for adler32 (adler32-zlib) [ 2046.369248] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [ 2046.535905] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 2048.143423] Key type lgssc registered [ 2048.906428] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2053.601862] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2053.612950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2054.859354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2057.605499] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2062.706710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2062.959328] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2065.673992] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2070.057291] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2073.399684] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2077.562409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2077.835256] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2079.925298] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2081.666740] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2086.858310] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2091.236255] Lustre: Failing over lustre-MDT0000 [ 2091.440562] Lustre: server umount lustre-MDT0000 complete [ 2093.541088] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2093.547526] Lustre: Skipped 2 previous similar messages [ 2093.552046] LustreError: 74774:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2093.556437] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2093.571813] LustreError: 74774:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2097.520749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2097.610564] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2097.705600] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2097.789491] LustreError: 77862:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2097.794696] LustreError: 77862:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.131@tcp: cfg command failed: rc = -9 [ 2097.799583] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2097.799583] [ 2097.805568] LustreError: MGC192.168.206.131@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2097.814490] LustreError: 77850:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2097.817713] LustreError: 77850:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -9 [ 2097.822772] LustreError: 77850:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 2097.827378] LustreError: 77850:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2097.848449] Lustre: server umount lustre-MDT0000 complete [ 2097.850605] LustreError: 77850:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2098.659363] LustreError: 75505:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2098.673894] LustreError: 75505:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2101.814214] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2101.917716] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2102.096685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2102.143126] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2103.858525] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2104.554860] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2107.371562] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2107.389774] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2107.420980] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2112.483660] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2112.484822] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2112.485819] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2112.493271] Lustre: Skipped 1 previous similar message [ 2115.603939] Lustre: server umount lustre-OST0000 complete [ 2122.721120] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2122.721362] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2122.727560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2122.734465] Lustre: Skipped 1 previous similar message [ 2122.743156] Lustre: Skipped 2 previous similar messages [ 2124.027904] Lustre: server umount lustre-MDT0000 complete [ 2126.219430] LustreError: 74758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789481835 with bad export cookie 18059425141378047955 [ 2126.231343] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2126.412114] Lustre: server umount lustre-MDT0001 complete [ 2133.581331] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 2135.492145] Key type lgssc unregistered [ 2135.701463] LNet: 79638:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2135.708647] LNetError: 79638:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2135.718728] LNet: Removed LNI 192.168.206.131@tcp [ 2136.233695] Key type .llcrypt unregistered [ 2136.235964] Key type ._llcrypt unregistered [ 2143.481920] Key type ._llcrypt registered [ 2143.483362] Key type .llcrypt registered [ 2143.564066] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 2143.689445] Key type .llcrypt unregistered [ 2143.691840] Key type ._llcrypt unregistered [ 2156.189100] Key type ._llcrypt registered [ 2156.190701] Key type .llcrypt registered [ 2156.260187] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 2156.721748] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2156.785061] alg: No test for adler32 (adler32-zlib) [ 2157.714114] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [ 2157.873968] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 2159.519187] Key type lgssc registered [ 2160.286731] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2164.604881] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2164.618133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2165.927820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2168.312723] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2172.864837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2173.089706] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2175.339922] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2179.050479] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2181.960611] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2185.437626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2185.634644] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2188.673866] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2190.765109] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2192.998604] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2203.107060] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2203.107246] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2203.116033] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2203.127630] Lustre: Skipped 1 previous similar message [ 2208.230722] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2208.235990] Lustre: Skipped 2 previous similar messages [ 2208.322971] Lustre: server umount lustre-OST0000 complete [ 2213.345565] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2213.349497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2213.360526] Lustre: Skipped 1 previous similar message [ 2216.631523] Lustre: server umount lustre-MDT0000 complete [ 2218.464103] LustreError: 82570:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2218.477939] LustreError: 82570:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2218.735290] LustreError: 81489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789481928 with bad export cookie 10543716206711927688 [ 2218.738354] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2218.744532] LustreError: 81489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2218.953985] Lustre: server umount lustre-MDT0001 complete [ 2224.706808] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 2226.347893] Key type lgssc unregistered [ 2226.562720] LNet: 85034:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2226.568745] LNetError: 85034:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2226.583100] LNet: Removed LNI 192.168.206.131@tcp [ 2227.097383] Key type .llcrypt unregistered [ 2227.099509] Key type ._llcrypt unregistered [ 2241.509250] Key type ._llcrypt registered [ 2241.510587] Key type .llcrypt registered [ 2241.587656] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 2242.116233] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2242.183432] alg: No test for adler32 (adler32-zlib) [ 2243.088169] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [ 2243.214471] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 2244.825362] Key type lgssc registered [ 2245.450933] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2249.467373] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2249.481041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2250.705028] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2252.769561] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2257.479142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2257.702660] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2260.443743] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2264.695493] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2267.701459] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2271.273567] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2271.494224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2274.891722] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2275.627467] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2279.115960] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2288.098261] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2288.098749] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2288.108042] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2288.116927] Lustre: Skipped 1 previous similar message [ 2289.909592] Lustre: server umount lustre-OST0000 complete [ 2292.183529] Lustre: server umount lustre-MDT0000 complete [ 2293.217739] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2293.218410] LustreError: 86421:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2293.225234] Lustre: Skipped 2 previous similar messages [ 2293.240334] LustreError: 86421:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2294.034800] LustreError: 86403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789482003 with bad export cookie 1613156417520127714 [ 2294.037615] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2294.044408] LustreError: 86403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2300.278340] Lustre: server umount lustre-MDT0001 complete [ 2305.496506] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 2307.005459] Key type lgssc unregistered [ 2307.149886] LNet: 89950:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2307.153810] LNetError: 89950:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2307.167942] LNet: Removed LNI 192.168.206.131@tcp [ 2307.595856] Key type .llcrypt unregistered [ 2307.599187] Key type ._llcrypt unregistered [ 2320.717971] Key type ._llcrypt registered [ 2320.719395] Key type .llcrypt registered [ 2320.797454] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 2321.263262] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2321.274625] alg: No test for adler32 (adler32-zlib) [ 2322.197212] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [ 2322.347138] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 2323.967244] Key type lgssc registered [ 2324.530482] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2328.413644] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2328.424975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2329.662801] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2331.900234] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2336.820871] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2337.039569] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2339.735655] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2344.421495] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2347.669048] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2352.065683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2352.340511] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2355.444286] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2356.019733] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2360.443994] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2371.041994] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2371.043258] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2371.046495] LustreError: Skipped 1 previous similar message [ 2371.057022] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2372.065172] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2372.068032] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2372.070486] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2372.083329] Lustre: Skipped 1 previous similar message [ 2373.096566] Lustre: server umount lustre-OST0000 complete [ 2377.185126] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2377.186406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2377.191572] Lustre: Skipped 1 previous similar message [ 2381.144949] Lustre: server umount lustre-MDT0000 complete [ 2382.304215] LustreError: 91336:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2382.315667] LustreError: 91336:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2383.223602] LustreError: 91316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789482092 with bad export cookie 13779726235159143414 [ 2383.226232] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2383.231185] LustreError: 91316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2383.414584] Lustre: server umount lustre-MDT0001 complete [ 2389.722446] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 2391.667938] Key type lgssc unregistered [ 2391.875514] LNet: 94866:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2391.878496] LNetError: 94866:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2391.888430] LNet: Removed LNI 192.168.206.131@tcp [ 2392.403123] Key type .llcrypt unregistered [ 2392.405172] Key type ._llcrypt unregistered [ 2405.193190] Key type ._llcrypt registered [ 2405.195619] Key type .llcrypt registered [ 2405.253774] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 2405.800276] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2405.818861] alg: No test for adler32 (adler32-zlib) [ 2406.752253] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [ 2406.895134] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 2408.511183] Key type lgssc registered [ 2409.300385] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2414.177749] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2414.205351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2429.727390] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2434.847343] LustreError: 96228:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.131@tcp: failed processing log, type 4: rc = -110 [ 2461.473430] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2465.463642] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2469.399722] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2472.565709] Lustre: server umount lustre-OST0000 complete [ 2476.755113] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 10:23:05 (1789482185) [ 2479.890799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2480.100422] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2482.041559] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2485.217281] LustreError: 97864:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2485.228243] LustreError: 97864:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2486.355047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2486.578361] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2488.903469] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2493.034645] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2496.172880] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2499.819292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2500.048643] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2503.718633] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2505.197351] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 2507.803325] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2520.544200] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2520.551552] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2520.566075] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2522.596784] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2522.600041] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2522.608976] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2525.316494] Lustre: server umount lustre-OST0000 complete [ 2527.712791] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2527.716700] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2527.724760] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2527.736663] Lustre: Skipped 1 previous similar message [ 2527.743303] Lustre: Skipped 2 previous similar messages [ 2532.833337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2532.841622] Lustre: Skipped 1 previous similar message [ 2533.574721] Lustre: server umount lustre-MDT0000 complete [ 2535.697595] LustreError: 99704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789482245 with bad export cookie 10013620295044361770 [ 2535.699557] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2535.707086] LustreError: 99704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2535.914385] Lustre: server umount lustre-MDT0001 complete [ 2542.308826] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 2543.869683] Key type lgssc unregistered [ 2544.116094] LNet: 101556:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2544.124598] LNetError: 101556:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2544.137577] LNet: Removed LNI 192.168.206.131@tcp [ 2544.738600] Key type .llcrypt unregistered [ 2544.741296] Key type ._llcrypt unregistered [ 2568.788822] Key type ._llcrypt registered [ 2568.790170] Key type .llcrypt registered [ 2568.865121] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 2569.759917] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2569.773181] alg: No test for adler32 (adler32-zlib) [ 2570.730776] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [ 2570.856514] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 2572.471221] Key type lgssc registered [ 2573.179375] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2577.912498] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2577.926719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2579.224418] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2581.836637] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2586.520876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2586.722748] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2588.663519] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2592.167369] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2594.938973] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2598.173545] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2598.330719] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2601.236755] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2604.454834] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 2604.553505] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2614.753347] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2614.758080] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2614.764964] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2616.802158] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2616.802530] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2616.809326] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2619.947236] Lustre: server umount lustre-OST0000 complete [ 2621.920535] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2621.921990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2621.930130] Lustre: Skipped 1 previous similar message [ 2621.941597] Lustre: Skipped 2 previous similar messages [ 2627.040707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2627.048565] Lustre: Skipped 1 previous similar message [ 2627.844160] Lustre: server umount lustre-MDT0000 complete [ 2629.853135] LustreError: 103694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789482339 with bad export cookie 4346581537227424398 [ 2629.856524] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2629.859570] LustreError: 103694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2630.009619] Lustre: server umount lustre-MDT0001 complete [ 2635.515319] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 2637.019581] Key type lgssc unregistered [ 2637.188916] LNet: 107243:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2637.194687] LNetError: 107243:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2637.212463] LNet: Removed LNI 192.168.206.131@tcp [ 2637.705122] Key type .llcrypt unregistered [ 2637.707294] Key type ._llcrypt unregistered [ 2647.396855] Key type ._llcrypt registered [ 2647.398990] Key type .llcrypt registered [ 2647.455364] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 2647.976875] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2648.020269] alg: No test for adler32 (adler32-zlib) [ 2648.928037] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [ 2649.082767] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 2650.687324] Key type lgssc registered [ 2651.291709] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2655.885973] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2655.900711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2671.392911] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2676.511375] LustreError: 108459:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.131@tcp: failed processing log, type 4: rc = -110 [ 2703.199294] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2706.354190] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2710.146914] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2713.608694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2713.783717] LustreError: MGC192.168.206.131@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2713.795318] LustreError: 109482:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2713.802167] LustreError: 109482:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 2713.806862] LustreError: 109482:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 2713.856628] Lustre: server umount lustre-MDT0000 complete [ 2713.859377] LustreError: 109482:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2728.489842] Lustre: server umount lustre-OST0000 complete [ 2733.137301] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_hostid [ 2738.207510] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 2756.894182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2757.014146] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2757.032375] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2757.076438] Lustre: lustre-MDT0000: new disk, initializing [ 2757.120212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2757.132478] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2759.118164] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2765.144717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2765.223575] Lustre: 112271:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2765.266833] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2765.270475] Lustre: Skipped 1 previous similar message [ 2765.320402] Lustre: lustre-MDT0001: new disk, initializing [ 2765.401977] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2765.432866] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2765.445337] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2767.778655] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2773.038851] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2775.800356] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2779.302504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2779.480056] Lustre: lustre-OST0000: new disk, initializing [ 2779.483172] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2779.488109] Lustre: 114238:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2779.542045] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2782.840581] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2785.295910] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2785.303370] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2785.358616] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2788.098507] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2790.453616] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2790.567268] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2792.891527] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2792.997864] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2795.492654] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2795.493461] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2795.504565] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2795.509983] Lustre: Skipped 1 previous similar message [ 2800.610350] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2800.616680] Lustre: Skipped 1 previous similar message [ 2805.739454] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2805.744559] Lustre: Skipped 1 previous similar message [ 2808.287134] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2808.461403] Lustre: server umount lustre-OST0000 complete [ 2810.849169] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2810.856240] Lustre: Skipped 1 previous similar message [ 2810.858297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2810.862230] Lustre: Skipped 1 previous similar message [ 2815.969468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2815.975021] Lustre: Skipped 1 previous similar message [ 2816.809817] Lustre: server umount lustre-MDT0000 complete [ 2818.895043] LustreError: 113064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789482528 with bad export cookie 1561406108449669454 [ 2818.896453] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2818.901335] LustreError: 113064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2819.067809] Lustre: server umount lustre-MDT0001 complete [ 2824.196592] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 10:28:52 (1789482532) [ 2826.301092] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 2835.987469] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_hostid [ 2841.756304] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 2861.762244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2861.949807] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2861.982876] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2862.030535] Lustre: lustre-MDT0000: new disk, initializing [ 2862.098659] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2862.120745] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2864.623275] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2871.350139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2871.457304] Lustre: 119140:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2871.499133] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2871.502377] Lustre: Skipped 1 previous similar message [ 2871.561625] Lustre: lustre-MDT0001: new disk, initializing [ 2871.645781] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2871.676083] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2871.681239] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2874.173547] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2880.226980] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2883.532755] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2887.505799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2887.656186] Lustre: lustre-OST0000: new disk, initializing [ 2887.659531] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2887.667812] Lustre: 121107:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2887.709478] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2891.247048] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2894.354773] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2894.360433] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2894.417620] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2896.795084] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2898.897831] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2899.003450] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2901.012824] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2901.108691] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2904.544216] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2904.547830] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2904.549112] LustreError: Skipped 2 previous similar messages [ 2904.559032] Lustre: Skipped 1 previous similar message [ 2904.563518] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2904.570198] Lustre: Skipped 1 previous similar message [ 2916.319223] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2916.429485] Lustre: server umount lustre-OST0000 complete [ 2919.907064] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2919.916257] Lustre: Skipped 1 previous similar message [ 2924.628140] Lustre: server umount lustre-MDT0000 complete [ 2926.485651] LustreError: 121106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789482635 with bad export cookie 1561406108449670336 [ 2926.488544] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2926.496635] LustreError: 121106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2926.688106] Lustre: server umount lustre-MDT0001 complete [ 2931.104759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2931.355725] LustreError: 123324:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2931.367906] LustreError: 123324:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2931.412888] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2933.800660] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2936.802421] LustreError: 123325:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2938.821886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2941.525600] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2945.813764] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2949.197454] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2952.826944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2956.434303] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 2960.558780] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2968.821869] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 2971.616156] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2971.619922] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2971.629538] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2971.633341] Lustre: Skipped 6 previous similar messages [ 2974.688102] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2977.320688] Lustre: server umount lustre-OST0000 complete [ 2979.810632] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2979.823601] Lustre: Skipped 2 previous similar messages [ 2985.856305] Lustre: server umount lustre-MDT0000 complete [ 2987.788257] LustreError: 123305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789482697 with bad export cookie 1561406108449671218 [ 2987.792879] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2987.794556] LustreError: 123305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2987.947457] Lustre: server umount lustre-MDT0001 complete [ 2993.165657] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 2994.641190] Key type lgssc unregistered [ 2994.843429] LNet: 126948:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2994.849884] LNetError: 126948:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2994.863695] LNet: Removed LNI 192.168.206.131@tcp [ 2995.309055] Key type .llcrypt unregistered [ 2995.312596] Key type ._llcrypt unregistered [ 3004.754056] Key type ._llcrypt registered [ 3004.756337] Key type .llcrypt registered [ 3004.823360] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_hostid [ 3013.305802] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 3013.911445] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3013.931223] alg: No test for adler32 (adler32-zlib) [ 3014.839341] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [ 3014.994564] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 3016.615248] Key type lgssc registered [ 3017.285300] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3038.423493] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3038.439530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3039.555265] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3039.572658] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3039.611887] Lustre: lustre-MDT0000: new disk, initializing [ 3039.651626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3039.661471] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3042.191702] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 3049.203458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3049.272887] Lustre: 130072:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3049.314886] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3049.317444] Lustre: Skipped 1 previous similar message [ 3049.368886] Lustre: lustre-MDT0001: new disk, initializing [ 3049.425565] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3049.454702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3049.465242] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3052.616522] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 3060.700075] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3064.355750] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3068.806120] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3068.959692] Lustre: lustre-OST0000: new disk, initializing [ 3068.962572] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3068.967327] Lustre: 132035:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3069.036563] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3073.291770] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 3079.190026] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3079.197731] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3079.287398] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3079.929612] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3082.723496] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3082.847757] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3085.262638] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3085.408381] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3089.379052] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3089.384079] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3089.388769] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3089.400481] Lustre: Skipped 1 previous similar message [ 3092.696790] Lustre: server umount lustre-OST0000 complete [ 3095.520705] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3095.525269] LustreError: Skipped 1 previous similar message [ 3095.527600] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3095.534633] Lustre: Skipped 1 previous similar message [ 3095.537550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3099.620020] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3099.620863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3099.634489] Lustre: Skipped 1 previous similar message [ 3104.736937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3108.832656] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3108.837079] Lustre: Skipped 2 previous similar messages [ 3109.344293] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3109.452205] Lustre: server umount lustre-MDT0000 complete [ 3111.474510] LustreError: 130065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789482820 with bad export cookie 9104514077829577959 [ 3111.479766] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3111.483372] LustreError: 130065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3111.710387] Lustre: server umount lustre-MDT0001 complete [ 3115.929461] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 10:33:44 (1789482824) [ 3119.268471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3119.536286] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3121.568958] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 3124.710191] LustreError: 134448:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3124.723160] LustreError: 134448:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3126.370854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3126.556432] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3129.226862] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 3133.935583] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3137.216848] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3138.406715] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3138.408130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3138.429380] Lustre: Skipped 1 previous similar message [ 3144.640627] Lustre: server umount lustre-MDT0000 complete [ 3146.840099] LustreError: 134427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789482856 with bad export cookie 9104514077829578841 [ 3146.843116] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3146.847739] LustreError: 134427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3147.055349] Lustre: server umount lustre-MDT0001 complete [ 3151.054951] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 10:34:19 (1789482859) [ 3154.114188] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3169.631517] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3174.751844] LustreError: 136860:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.131@tcp: failed processing log, type 4: rc = -110 [ 3200.415405] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3204.592602] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 3209.069191] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3210.236642] Lustre: server umount lustre-OST0000 complete [ 3215.986872] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 10:35:24 (1789482924) [ 3220.518734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3220.805711] LustreError: 138305:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3220.860702] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3224.047300] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 3226.086233] LustreError: 138306:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3229.534680] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3232.880269] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 3237.506183] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3240.786662] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3244.701776] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3248.527691] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 3252.578184] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3265.504453] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3265.508932] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3265.519058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3265.523836] Lustre: Skipped 5 previous similar messages [ 3267.210925] Lustre: server umount lustre-MDT0000 complete [ 3269.224991] LustreError: 138285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789482978 with bad export cookie 9104514077829579506 [ 3269.226857] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3269.235953] LustreError: 138285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3269.419099] Lustre: server umount lustre-MDT0001 complete [ 3280.719387] Lustre: server umount lustre-OST0000 complete [ 3285.577879] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 10:36:34 (1789482994) [ 3289.419835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3289.653861] LustreError: 141953:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3289.688552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3289.692367] Lustre: Skipped 2 previous similar messages [ 3291.890370] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 3294.690934] LustreError: 141954:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3296.325423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3298.628763] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 3302.373598] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3305.268584] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3309.046271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3312.549800] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 3315.378976] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 3316.506396] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3318.809322] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3318.913623] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3321.179590] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3322.358093] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 1 sec [ 3325.921376] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3325.925973] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3325.934818] Lustre: Skipped 2 previous similar messages [ 3325.938750] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3329.587126] Lustre: server umount lustre-OST0000 complete [ 3337.666813] Lustre: server umount lustre-MDT0000 complete [ 3339.623975] LustreError: 141936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789483049 with bad export cookie 9104514077829580556 [ 3339.626205] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3339.633426] LustreError: 141936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3339.802783] Lustre: server umount lustre-MDT0001 complete [ 3344.024443] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 10:37:32 (1789483052) [ 3347.666383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3363.167550] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3368.287486] LustreError: 146112:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.131@tcp: failed processing log, type 4: rc = -110 [ 3393.954768] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3393.962165] Lustre: Skipped 2 previous similar messages [ 3398.215403] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 3402.815576] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3406.501271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3406.732114] LustreError: 146137:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3406.827155] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 3409.500401] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 3415.184194] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3418.450465] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 3422.792780] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3425.876625] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3428.316766] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3428.435839] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3430.650465] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3430.753400] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3436.006039] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3436.006354] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3436.019504] Lustre: Skipped 3 previous similar messages [ 3436.021515] LustreError: Skipped 1 previous similar message [ 3436.026418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3436.029712] Lustre: Skipped 8 previous similar messages [ 3437.800144] Lustre: server umount lustre-MDT0000 complete [ 3439.937911] LustreError: 146120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789483149 with bad export cookie 9104514077829581585 [ 3439.941230] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3439.944986] LustreError: 146120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3440.109715] Lustre: server umount lustre-MDT0001 complete [ 3452.349044] Lustre: server umount lustre-OST0000 complete [ 3457.385916] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 10:39:25 (1789483165) [ 3461.353640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3476.831592] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3481.951568] LustreError: 150301:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.131@tcp: failed processing log, type 4: rc = -110 [ 3512.739283] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 3517.987647] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3522.177140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3522.383372] LustreError: 150326:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3522.397370] LustreError: 150326:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 3522.437504] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3522.445393] Lustre: Skipped 3 previous similar messages [ 3522.511250] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 3525.005266] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 3530.101517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3532.573212] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 3536.913981] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3540.383278] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3544.581028] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3544.720784] Lustre: lustre-OST0001: new disk, initializing [ 3544.723374] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3544.728480] Lustre: 153168:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3548.574884] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 3553.808781] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3553.816646] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3553.873849] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3554.370134] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3556.841902] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3556.973335] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3559.750532] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3559.920795] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3563.492872] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3563.501323] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3563.509348] Lustre: Skipped 2 previous similar messages [ 3567.187477] Lustre: server umount lustre-OST0000 complete [ 3569.123781] LustreError: 150317:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3569.142464] LustreError: 150317:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3574.247372] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3574.250602] Lustre: Skipped 3 previous similar messages [ 3575.538315] Lustre: server umount lustre-OST0001 complete [ 3584.330109] Lustre: server umount lustre-MDT0000 complete [ 3586.641635] LustreError: 150308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789483296 with bad export cookie 9104514077829582537 [ 3586.645204] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3586.652391] LustreError: 150308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3586.879425] Lustre: server umount lustre-MDT0001 complete [ 3603.735178] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 10:41:52 (1789483312) [ 3604.787143] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 3605.829354] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 3606.833803] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 10:41:55 (1789483315) [ 3611.576758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3611.770949] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3611.798338] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3615.154251] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 3620.689225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3620.775468] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3620.809665] Lustre: 156730:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3623.627971] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 3628.129469] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3631.891491] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3635.889610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3635.999685] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3637.266240] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 3640.021457] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 3644.690893] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3647.580588] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3647.735442] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3650.581200] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3650.744857] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3652.578246] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3652.586093] LustreError: Skipped 2 previous similar messages [ 3652.588577] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3652.610867] Lustre: Skipped 6 previous similar messages [ 3666.399126] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3666.481261] Lustre: server umount lustre-OST0000 complete [ 3683.183954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3686.723526] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 3691.088541] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3692.586911] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3692.599866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3692.607716] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3692.622802] Lustre: Skipped 1 previous similar message [ 3694.112428] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3694.223768] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3696.522956] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3696.642872] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3699.857520] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff918c51f1d800.ost_server_uuid 50 [ 3700.761358] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff918c51f1d800.ost_server_uuid in FULL state after 0 sec [ 3700.864371] LustreError: 159116:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.206.31@tcp) returned error from blocking AST (req@000000001e0997f7 x1876408637016192 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000141c503f/0x7e59bb50e5935823 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.206.31@tcp remote: 0xed1808199289cb75 expref: 5 pid: 159875 timeout: 3800 lvb_type: 0 lru_score: 0 lru_type: 0 [ 3700.888626] LustreError: lustre-MDT0000: A client on nid 192.168.206.31@tcp was evicted due to a lock blocking callback time out: rc -107 [ 3700.895986] LustreError: 156725:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.206.31@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000141c503f/0x7e59bb50e5935823 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.206.31@tcp remote: 0xed1808199289cb75 expref: 6 pid: 159875 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 3706.856413] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3709.016496] Lustre: server umount lustre-OST0000 complete [ 3717.517821] Lustre: server umount lustre-MDT0000 complete [ 3718.116326] LustreError: 156740:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3718.127029] LustreError: 156740:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 3719.562619] LustreError: 158587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789483429 with bad export cookie 9104514077829583720 [ 3719.567176] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3719.570731] LustreError: 158587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3725.807227] Lustre: server umount lustre-MDT0001 complete [ 3731.824460] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 3733.424366] Key type lgssc unregistered [ 3733.649673] LNet: 162834:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3733.653078] LNetError: 162834:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3733.664462] LNet: Removed LNI 192.168.206.131@tcp [ 3734.140435] Key type .llcrypt unregistered [ 3734.142768] Key type ._llcrypt unregistered [ 3748.030684] Key type ._llcrypt registered [ 3748.032673] Key type .llcrypt registered [ 3748.102037] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 3748.703286] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3748.736516] alg: No test for adler32 (adler32-zlib) [ 3749.654381] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [ 3749.819924] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 3751.463316] Key type lgssc registered [ 3752.242619] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3757.223466] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3757.238132] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3758.573075] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3761.376801] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 3766.371390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3766.603849] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3769.045862] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 3773.191619] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3776.483497] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3780.637141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3780.933437] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3783.988293] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 3784.574413] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 3789.088238] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3796.608517] Lustre: Failing over lustre-MDT0000 [ 3796.787543] Lustre: server umount lustre-MDT0000 complete [ 3797.475267] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3797.475287] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3797.476267] LustreError: 164215:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3797.476276] LustreError: 164215:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3797.485138] Lustre: Skipped 1 previous similar message [ 3802.594125] LustreError: 164215:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3802.606154] LustreError: 164215:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 3807.713716] LustreError: 164221:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3807.734115] LustreError: 164221:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3808.120141] LustreError: 164200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.206.31@tcp arrived at 1789483517 with bad export cookie 6062725635092973285 [ 3812.326582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3812.403414] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3812.566321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3812.607986] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3815.105523] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 3816.408380] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3816.417470] Lustre: lustre-MDT0000: Denying connection for new client c6970bba-2fce-4d88-b551-ad2bd20320ed (at 192.168.206.31@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 3817.960823] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3828.193626] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3828.194819] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3828.209518] Lustre: Skipped 1 previous similar message [ 3828.213522] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3828.217275] Lustre: Skipped 1 previous similar message [ 3829.885232] Lustre: server umount lustre-OST0000 complete [ 3832.199928] LustreError: 167944:0:(ldlm_lib.c:3038:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3832.204613] Lustre: 167294:0:(ldlm_lib.c:2438:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3832.209294] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 3832.219744] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 3832.225189] LustreError: 167294:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000a569e15d x1876409407542016/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3832.234741] LustreError: 167294:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 3832.238902] LustreError: 167294:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 3832.243523] Lustre: lustre-MDT0000: Recovery over after 0:16, of 2 clients 0 recovered and 2 were evicted. [ 3833.314689] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3833.336927] Lustre: Skipped 1 previous similar message [ 3833.341137] LustreError: 166706:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3834.437702] Lustre: server umount lustre-MDT0000 complete [ 3836.688924] LustreError: 164200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789483546 with bad export cookie 6062725635092973390 [ 3836.691713] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3836.695481] LustreError: 164200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3836.707087] LustreError: 163529:0:(import.c:752:ptlrpc_connect_import_locked()) can't connect to a closed import [ 3838.892935] Lustre: server umount lustre-MDT0001 complete [ 3844.594497] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 3846.184416] Key type lgssc unregistered [ 3846.388061] LNet: 168697:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3846.393667] LNetError: 168697:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3847.463867] LNet: Removed LNI 192.168.206.131@tcp [ 3847.970833] Key type .llcrypt unregistered [ 3847.972993] Key type ._llcrypt unregistered [ 3861.386780] Key type ._llcrypt registered [ 3861.388699] Key type .llcrypt registered [ 3861.447635] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 3862.014534] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3862.034872] alg: No test for adler32 (adler32-zlib) [ 3862.989459] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [ 3863.147414] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 3864.759250] Key type lgssc registered [ 3865.504541] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3870.160634] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3870.180979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3871.468596] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3873.778918] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 3878.421836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3878.632736] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3880.731751] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 3884.905102] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3888.419657] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3892.281531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3892.461069] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3895.539742] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 3897.251893] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 3902.544870] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3909.089950] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3909.095090] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3909.105445] Lustre: Skipped 1 previous similar message [ 3913.600812] Lustre: server umount lustre-OST0000 complete [ 3919.329580] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3919.330149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3919.334813] Lustre: Skipped 1 previous similar message [ 3919.342575] Lustre: Skipped 2 previous similar messages [ 3924.448840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3924.453524] Lustre: Skipped 1 previous similar message [ 3928.545992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3930.079155] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3932.230501] Lustre: server umount lustre-MDT0000 complete [ 3933.664683] LustreError: 171061:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3933.674891] LustreError: 171061:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 3934.194135] LustreError: 170075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789483643 with bad export cookie 16221689098339173378 [ 3934.200372] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3934.203904] LustreError: 170075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3934.376990] Lustre: server umount lustre-MDT0001 complete [ 3939.573408] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 3941.186882] Key type lgssc unregistered [ 3941.381988] LNet: 173621:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3941.388453] LNetError: 173621:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3941.401748] LNet: Removed LNI 192.168.206.131@tcp [ 3941.862946] Key type .llcrypt unregistered [ 3941.864731] Key type ._llcrypt unregistered [ 3960.007196] Key type ._llcrypt registered [ 3960.008924] Key type .llcrypt registered [ 3960.081231] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 3960.577497] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3960.644668] alg: No test for adler32 (adler32-zlib) [ 3961.605584] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [ 3961.757158] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 3963.367218] Key type lgssc registered [ 3964.100225] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3968.872331] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3968.891930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3970.169496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3972.673544] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 3977.869853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3978.137026] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3980.683326] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 3984.898273] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3988.358292] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3992.556800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3992.778891] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3994.877835] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 3997.896821] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 4003.759859] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4008.416930] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4008.424364] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4008.425678] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4008.432542] Lustre: Skipped 1 previous similar message [ 4010.139450] Lustre: server umount lustre-OST0000 complete [ 4012.398089] Lustre: server umount lustre-MDT0000 complete [ 4013.538846] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4013.551268] LustreError: 175291:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4013.563750] LustreError: 175291:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4014.512718] LustreError: 175277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789483723 with bad export cookie 16829801424983770897 [ 4014.513676] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4014.519487] LustreError: 175277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4014.693697] Lustre: server umount lustre-MDT0001 complete [ 4020.804958] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 4022.664739] Key type lgssc unregistered [ 4022.840726] LNet: 178819:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4022.849837] LNetError: 178819:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4022.862100] LNet: Removed LNI 192.168.206.131@tcp [ 4023.398487] Key type .llcrypt unregistered [ 4023.401234] Key type ._llcrypt unregistered [ 4036.939934] Key type ._llcrypt registered [ 4036.942484] Key type .llcrypt registered [ 4037.006244] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 4037.575842] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4037.648514] alg: No test for adler32 (adler32-zlib) [ 4038.586140] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [ 4038.737470] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 4040.343310] Key type lgssc registered [ 4041.011305] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4046.860411] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4046.873420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4048.076896] Lustre: *** cfs_fail_loc=135, val=0*** [ 4048.126964] LustreError: 180316:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 4048.129692] LustreError: 180316:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.131@tcp: cfg command failed: rc = -2 [ 4048.133495] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 4048.133495] [ 4048.137309] LustreError: MGC192.168.206.131@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 4048.143182] LustreError: 180277:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 4048.147608] LustreError: 180277:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 4048.155064] LustreError: 180277:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 4048.160958] LustreError: 180277:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 4048.211741] Lustre: server umount lustre-MDT0000 complete [ 4048.214347] LustreError: 180277:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4051.749636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4051.966310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4054.255240] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 4059.261353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4059.523937] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4062.235710] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 4066.320667] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4069.405772] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4073.952822] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4073.955080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4073.959454] Lustre: Skipped 1 previous similar message [ 4076.725472] Lustre: server umount lustre-MDT0000 complete [ 4078.421599] LustreError: 180667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789483787 with bad export cookie 3247364831490528891 [ 4078.425198] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4078.428809] LustreError: 180667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4078.606531] Lustre: server umount lustre-MDT0001 complete [ 4084.415689] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 4085.872851] Key type lgssc unregistered [ 4086.061680] LNet: 183004:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4086.065284] LNetError: 183004:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4086.077475] LNet: Removed LNI 192.168.206.131@tcp [ 4086.543815] Key type .llcrypt unregistered [ 4086.545652] Key type ._llcrypt unregistered [ 4098.526535] Key type ._llcrypt registered [ 4098.528637] Key type .llcrypt registered [ 4098.607173] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 4098.798624] Key type .llcrypt unregistered [ 4098.800933] Key type ._llcrypt unregistered [ 4109.545778] Key type ._llcrypt registered [ 4109.548731] Key type .llcrypt registered [ 4109.626466] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 4110.278745] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4110.293684] alg: No test for adler32 (adler32-zlib) [ 4111.223176] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [ 4111.393577] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 4113.024942] Key type lgssc registered [ 4113.714804] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4118.245047] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4118.263053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4133.728527] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4138.848057] LustreError: 185104:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.131@tcp: failed processing log, type 4: rc = -110 [ 4165.471181] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4168.543075] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 4172.346796] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4175.951291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4176.218558] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4176.262099] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 4178.306305] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 4182.447925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4182.648477] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4184.814946] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 4188.801077] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4191.784063] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4193.431612] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 4201.954566] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4201.958469] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4201.966181] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4202.977270] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4202.981605] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4202.989418] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4205.424437] Lustre: server umount lustre-OST0000 complete [ 4208.098489] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4208.104559] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4208.114299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4208.117973] Lustre: Skipped 2 previous similar messages [ 4213.610630] Lustre: server umount lustre-MDT0000 complete [ 4213.728356] LustreError: 186147:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4213.737306] LustreError: 186147:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 4215.556512] LustreError: 185110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789483925 with bad export cookie 9844406094896035883 [ 4215.558705] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4215.565058] LustreError: 185110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4215.735928] Lustre: server umount lustre-MDT0001 complete [ 4220.794172] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 4222.276897] Key type lgssc unregistered [ 4222.466523] LNet: 189017:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4222.471191] LNetError: 189017:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4222.479601] LNet: Removed LNI 192.168.206.131@tcp [ 4222.904411] Key type .llcrypt unregistered [ 4222.906644] Key type ._llcrypt unregistered [ 4235.940072] Key type ._llcrypt registered [ 4235.943261] Key type .llcrypt registered [ 4236.003566] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 4236.603993] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4236.680798] alg: No test for adler32 (adler32-zlib) [ 4237.620418] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [ 4237.767809] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 4239.383340] Key type lgssc registered [ 4240.240139] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4245.518631] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4245.534990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4246.815376] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4249.538183] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 4255.310125] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4255.625439] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4258.735283] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 4263.520663] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4266.830242] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4270.682111] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4270.970947] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4271.985106] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 4274.295846] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 4278.341148] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4281.360712] Lustre: Failing over lustre-MDT0000 [ 4281.522962] Lustre: server umount lustre-MDT0000 complete [ 4284.987138] LustreError: 190399:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4285.000122] LustreError: 190399:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4286.432989] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4286.436587] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4286.436712] LustreError: 190401:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4286.445350] Lustre: Skipped 2 previous similar messages [ 4290.095226] LustreError: 190399:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4290.111163] LustreError: 190399:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4295.212263] LustreError: 190401:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4295.225692] LustreError: 190401:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4296.150765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4296.225676] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4296.370632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4296.400744] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4298.460847] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 4300.332403] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4300.926492] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 4301.803711] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4301.826974] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4301.865865] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 4306.635942] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 4312.040950] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4312.041338] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4312.042512] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4312.054428] Lustre: Skipped 1 previous similar message [ 4316.351510] Lustre: server umount lustre-OST0000 complete [ 4322.272738] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4322.273088] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4322.275923] LustreError: Skipped 1 previous similar message [ 4322.282301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4322.286280] Lustre: Skipped 2 previous similar messages [ 4324.661283] Lustre: server umount lustre-MDT0000 complete [ 4326.417418] LustreError: 190386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789484035 with bad export cookie 13891168890129574503 [ 4326.421431] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4326.424373] LustreError: 190386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4326.579371] Lustre: server umount lustre-MDT0001 complete [ 4331.834356] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 4333.352091] Key type lgssc unregistered [ 4333.531336] LNet: 195280:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4333.534290] LNetError: 195280:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4333.542535] LNet: Removed LNI 192.168.206.131@tcp [ 4334.004809] Key type .llcrypt unregistered [ 4334.006795] Key type ._llcrypt unregistered [ 4346.778671] Key type ._llcrypt registered [ 4346.780518] Key type .llcrypt registered [ 4346.831718] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 4347.275874] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4347.345893] alg: No test for adler32 (adler32-zlib) [ 4348.276057] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [ 4348.414870] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 4350.015231] Key type lgssc registered [ 4350.587622] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4354.601553] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4354.612562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4355.839434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4357.923874] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 4362.233556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4362.430224] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4364.617639] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 4368.621426] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4371.496292] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4374.971535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4375.164979] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4378.556705] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 4381.366916] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 4382.855259] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4385.881645] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 4389.702188] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 4402.858719] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 4407.264413] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4407.269265] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4407.277316] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4408.289135] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4408.292612] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4408.298693] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4412.838156] Lustre: server umount lustre-OST0000 complete [ 4418.538722] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4418.540603] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4418.549368] Lustre: Skipped 1 previous similar message [ 4418.559095] Lustre: Skipped 2 previous similar messages [ 4423.650823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4423.658782] Lustre: Skipped 1 previous similar message [ 4427.748229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4427.753162] Lustre: Skipped 1 previous similar message [ 4429.279406] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4429.448921] Lustre: server umount lustre-MDT0000 complete [ 4432.202054] LustreError: 196647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789484141 with bad export cookie 4423685652879746135 [ 4432.203210] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4432.211297] LustreError: 196647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4432.458108] Lustre: server umount lustre-MDT0001 complete [ 4439.144506] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 4440.801771] Key type lgssc unregistered [ 4441.019338] LNet: 200528:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4441.026077] LNetError: 200528:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4441.043380] LNet: Removed LNI 192.168.206.131@tcp [ 4441.570863] Key type .llcrypt unregistered [ 4441.573207] Key type ._llcrypt unregistered [ 4452.864426] Key type ._llcrypt registered [ 4452.866018] Key type .llcrypt registered [ 4452.930429] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 4453.426675] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4453.489991] alg: No test for adler32 (adler32-zlib) [ 4454.432262] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [ 4454.616394] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 4456.263272] Key type lgssc registered [ 4456.932905] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4461.521686] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4461.534247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4462.799039] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4465.574967] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 4468.196747] LustreError: 201767:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4468.207441] LustreError: 201767:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4470.337573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4470.567377] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4473.233334] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 4478.171646] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4482.245172] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4487.779451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4487.908125] LustreError: 203587:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4488.043719] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4492.234986] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 4493.303275] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 4497.113412] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4501.902102] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 4504.315580] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 4511.746770] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 4515.633260] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 4524.002660] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4524.009622] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4524.022236] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4527.072971] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4527.077588] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4527.082637] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4529.411929] Lustre: server umount lustre-OST0000 complete [ 4532.193740] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4532.195155] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4532.196069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4532.196073] Lustre: Skipped 1 previous similar message [ 4532.227210] Lustre: Skipped 1 previous similar message [ 4537.312484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4537.317831] Lustre: Skipped 2 previous similar messages [ 4538.088367] Lustre: server umount lustre-MDT0000 complete [ 4540.187947] LustreError: 201749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789484249 with bad export cookie 13427002697696934656 [ 4540.195612] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4540.196075] LustreError: 201749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4540.374243] Lustre: server umount lustre-MDT0001 complete [ 4546.704607] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 4548.497464] Key type lgssc unregistered [ 4548.717665] LNet: 206544:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4548.724568] LNetError: 206544:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4549.797756] LNet: Removed LNI 192.168.206.131@tcp [ 4550.340639] Key type .llcrypt unregistered [ 4550.342677] Key type ._llcrypt unregistered [ 4567.127356] Key type ._llcrypt registered [ 4567.129789] Key type .llcrypt registered [ 4567.208670] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 4567.828910] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4567.865989] alg: No test for adler32 (adler32-zlib) [ 4568.820216] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [ 4569.002095] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 4570.631717] Key type lgssc registered [ 4571.394970] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4576.488584] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4576.501251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4577.747634] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4580.583767] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 4586.060571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4586.350688] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4588.936791] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 4593.514352] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4597.026273] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4601.032768] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4601.334221] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4603.369170] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 4605.072519] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 4610.541257] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4620.777747] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4620.777904] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4620.796316] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4625.889947] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4625.893288] Lustre: Skipped 2 previous similar messages [ 4626.565467] Lustre: server umount lustre-OST0000 complete [ 4631.009805] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4631.018297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4631.022934] Lustre: Skipped 2 previous similar messages [ 4631.033703] Lustre: Skipped 1 previous similar message [ 4635.287118] Lustre: server umount lustre-MDT0000 complete [ 4636.130403] LustreError: 208659:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4636.142074] LustreError: 208659:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4637.400731] LustreError: 207912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789484346 with bad export cookie 12215393850753405572 [ 4637.405434] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4637.409804] LustreError: 207912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4637.661618] Lustre: server umount lustre-MDT0001 complete [ 4644.071628] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 4645.845374] Key type lgssc unregistered [ 4646.044659] LNet: 211612:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4646.052856] LNetError: 211612:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4646.071601] LNet: Removed LNI 192.168.206.131@tcp [ 4646.580432] Key type .llcrypt unregistered [ 4646.582925] Key type ._llcrypt unregistered [ 4658.294369] Key type ._llcrypt registered [ 4658.296216] Key type .llcrypt registered [ 4658.355592] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 4658.929592] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4658.974202] alg: No test for adler32 (adler32-zlib) [ 4659.911630] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [ 4660.069312] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 4661.671238] Key type lgssc registered [ 4662.382732] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4667.097515] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4667.109813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4668.366479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4671.028196] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 4676.389503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4676.626681] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4679.537358] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 4684.424675] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4687.790249] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4692.250579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4692.557389] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4693.610527] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 4696.731788] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 4701.801609] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4712.418941] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4712.420761] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4712.436886] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4712.446411] Lustre: Skipped 1 previous similar message [ 4716.914380] Lustre: server umount lustre-OST0000 complete [ 4722.657170] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4722.657706] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4722.668092] Lustre: Skipped 1 previous similar message [ 4722.669437] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4722.680276] Lustre: Skipped 1 previous similar message [ 4727.776269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4727.781856] Lustre: Skipped 1 previous similar message [ 4732.896577] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4732.899420] Lustre: Skipped 1 previous similar message [ 4733.407121] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4733.507901] Lustre: server umount lustre-MDT0000 complete [ 4735.635430] LustreError: 215781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789484445 with bad export cookie 8203553886281325257 [ 4735.636971] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4735.648514] LustreError: 215781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4735.817419] Lustre: server umount lustre-MDT0001 complete [ 4741.755122] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 4743.296989] Key type lgssc unregistered [ 4743.486457] LNet: 216532:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4743.496117] LNetError: 216532:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4743.510485] LNet: Removed LNI 192.168.206.131@tcp [ 4744.009273] Key type .llcrypt unregistered [ 4744.011383] Key type ._llcrypt unregistered [ 4755.247180] Key type ._llcrypt registered [ 4755.249141] Key type .llcrypt registered [ 4755.320492] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 4755.974908] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4755.994179] alg: No test for adler32 (adler32-zlib) [ 4756.978287] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [ 4757.130938] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 4758.784695] Key type lgssc registered [ 4759.501614] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4764.378695] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4764.391976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4765.719231] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4768.340650] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 4773.198847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4773.422418] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4775.702345] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 4779.822636] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4782.863124] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4786.627369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4786.830671] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4790.038923] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 4790.956730] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 4794.017976] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4800.088958] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4800.138609] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4800.251480] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4804.163880] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 4807.922491] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4808.760816] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4819.710611] Lustre: Permanently deactivating lustre-OST0001 [ 4819.729253] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 4829.229173] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 4834.236409] Lustre: Permanently reactivating lustre-OST0001 [ 4834.248047] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 4834.250616] Lustre: Skipped 2 previous similar messages [ 4837.855790] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4837.869056] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 4837.874973] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4837.890089] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4837.915856] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4847.599664] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4847.605528] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4847.615257] Lustre: Skipped 2 previous similar messages [ 4850.561576] Lustre: server umount lustre-OST0001 complete [ 4852.711355] LustreError: 219606:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4852.737422] LustreError: 219606:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4854.752693] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4854.762307] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4854.778364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4854.785064] Lustre: Skipped 1 previous similar message [ 4857.828579] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4857.833835] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4857.834868] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4857.840282] LustreError: 221417:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4862.946083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4862.947470] LustreError: 221417:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4862.949912] Lustre: Skipped 2 previous similar messages [ 4862.962187] LustreError: 221417:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4868.063222] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4868.065861] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4868.066655] LustreError: 219605:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4868.079811] Lustre: Skipped 1 previous similar message [ 4868.142605] Lustre: server umount lustre-OST0000 complete [ 4870.624327] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4870.628361] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4876.287364] Lustre: server umount lustre-MDT0000 complete [ 4878.304409] LustreError: 217784:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4878.314966] LustreError: 217784:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4878.360626] LustreError: 220620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789484587 with bad export cookie 1836668126928215180 [ 4878.366265] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4878.368616] LustreError: 220620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4878.576380] Lustre: server umount lustre-MDT0001 complete [ 4884.706733] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 4886.369552] Key type lgssc unregistered [ 4886.531589] LNet: 222848:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4886.536786] LNetError: 222848:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4886.552181] LNet: Removed LNI 192.168.206.131@tcp [ 4887.020382] Key type .llcrypt unregistered [ 4887.022683] Key type ._llcrypt unregistered [ 4908.315521] Key type ._llcrypt registered [ 4908.317925] Key type .llcrypt registered [ 4908.391098] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 4908.977374] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4908.993609] alg: No test for adler32 (adler32-zlib) [ 4909.901505] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [ 4910.069827] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 4911.679155] Key type lgssc registered [ 4912.388377] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4916.949400] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4916.960718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4918.138966] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4918.155253] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4918.269195] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4920.922860] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 4925.978115] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4926.042424] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4926.066884] Lustre: 225026:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4926.214949] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4928.682858] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 4933.258632] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4936.415388] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4940.241186] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4940.335127] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4940.432377] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4943.445343] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 4947.193995] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4950.069272] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 4954.331962] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 4965.100682] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 4965.105896] Lustre: Skipped 1 previous similar message [ 4974.032100] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 4983.020635] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 4992.101216] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5011.681445] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 5011.685784] Lustre: Skipped 2 previous similar messages [ 5047.472108] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 5047.476090] Lustre: Skipped 3 previous similar messages [ 5113.533059] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 5113.536888] Lustre: Skipped 7 previous similar messages [ 5124.067153] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5124.067335] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5124.070604] LustreError: Skipped 1 previous similar message [ 5124.077116] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5124.083874] Lustre: Skipped 1 previous similar message [ 5129.185932] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5129.189258] Lustre: Skipped 1 previous similar message [ 5130.155824] Lustre: server umount lustre-OST0000 complete [ 5134.304848] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5134.307987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5134.316319] Lustre: Skipped 2 previous similar messages [ 5138.299194] Lustre: server umount lustre-MDT0000 complete [ 5140.190736] LustreError: 225018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789484849 with bad export cookie 1011732245373311399 [ 5140.194273] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5140.198694] LustreError: 225018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5140.366575] Lustre: server umount lustre-MDT0001 complete [ 5146.090326] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 5147.579640] Key type lgssc unregistered [ 5147.755474] LNet: 229825:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5147.764935] LNetError: 229825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5147.773581] LNet: Removed LNI 192.168.206.131@tcp [ 5148.252315] Key type .llcrypt unregistered [ 5148.254461] Key type ._llcrypt unregistered [ 5160.811248] Key type ._llcrypt registered [ 5160.812579] Key type .llcrypt registered [ 5160.877305] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 5161.483217] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5161.494901] alg: No test for adler32 (adler32-zlib) [ 5162.421666] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [ 5162.559292] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 5164.183333] Key type lgssc registered [ 5164.792422] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5169.183398] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5169.198332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5170.540885] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5172.981323] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 5177.571175] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5177.786216] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5180.065377] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 5184.375102] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5187.536623] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5191.460763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5191.704706] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5195.101039] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug -1 all [ 5195.816787] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 5199.320652] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5211.103996] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5211.108066] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5211.122133] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5212.198028] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5212.204883] Lustre: Skipped 2 previous similar messages [ 5217.125584] Lustre: server umount lustre-OST0000 complete [ 5223.399072] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5223.403866] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5223.407036] Lustre: Skipped 2 previous similar messages [ 5223.415330] Lustre: Skipped 1 previous similar message [ 5225.082517] Lustre: server umount lustre-MDT0000 complete [ 5227.014262] LustreError: 233789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789484936 with bad export cookie 12771949770271532446 [ 5227.018257] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5227.019684] LustreError: 233789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5227.193631] Lustre: server umount lustre-MDT0001 complete [ 5232.477534] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 5233.907339] Key type lgssc unregistered [ 5234.073301] LNet: 234899:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5234.079386] LNetError: 234899:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5234.091701] LNet: Removed LNI 192.168.206.131@tcp [ 5234.579824] Key type .llcrypt unregistered [ 5234.581614] Key type ._llcrypt unregistered [ 5246.749923] Key type ._llcrypt registered [ 5246.751711] Key type .llcrypt registered [ 5246.820340] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 5247.242485] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5247.309353] alg: No test for adler32 (adler32-zlib) [ 5248.240752] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [ 5248.393835] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 5250.007398] Key type lgssc registered [ 5250.592333] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5521.368838] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 5523.580445] Key type lgssc unregistered [ 5523.812225] LNet: 236904:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5523.816985] LNetError: 236904:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5523.831224] LNet: Removed LNI 192.168.206.131@tcp [ 5524.381138] Key type .llcrypt unregistered [ 5524.391224] Key type ._llcrypt unregistered [ 5539.273427] Key type ._llcrypt registered [ 5539.275477] Key type .llcrypt registered [ 5539.347647] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_hostid [ 5549.392769] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 5550.104827] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5550.118883] alg: No test for adler32 (adler32-zlib) [ 5551.103192] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [ 5551.292982] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 5552.935224] Key type lgssc registered [ 5553.716215] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5570.839423] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 5278 sec ========== 11:14:39 (1789485279) [ 5571.720244] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 11:14:40 (1789485280) === [ 5573.513942] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 11:14:42 (1789485282) === [ 5593.546156] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 5595.162898] Key type lgssc unregistered [ 5595.345117] LNet: 242109:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5595.350238] LNetError: 242109:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5595.365171] LNet: Removed LNI 192.168.206.131@tcp [ 5595.868129] Key type .llcrypt unregistered [ 5595.870415] Key type ._llcrypt unregistered