[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 507769929 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003279] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005011] kvm-guest: setup PV IPIs [ 0.008584] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.009026] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.010017] pid_max: default: 32768 minimum: 301 [ 0.011156] LSM: Security Framework initializing [ 0.012058] Yama: becoming mindful. [ 0.013039] SELinux: Initializing. [ 0.014062] *** VALIDATE selinux *** [ 0.022836] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027724] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029108] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030154] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031147] *** VALIDATE tmpfs *** [ 0.032493] *** VALIDATE proc *** [ 0.034073] *** VALIDATE cgroup *** [ 0.035011] *** VALIDATE cgroup2 *** [ 0.037171] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038170] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040028] Spectre V2 : User space: Vulnerable [ 0.041013] Speculative Store Bypass: Vulnerable [ 0.044396] debug: unmapping init [mem 0xffffffffa2a59000-0xffffffffa2a60fff] [ 0.047256] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048892] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049040] ... version: 2 [ 0.050030] ... bit width: 48 [ 0.051023] ... generic registers: 4 [ 0.052019] ... value mask: 0000ffffffffffff [ 0.053018] ... max period: 00007fffffffffff [ 0.054021] ... fixed-purpose events: 3 [ 0.055014] ... event mask: 000000070000000f [ 0.056349] rcu: Hierarchical SRCU implementation. [ 0.058673] smp: Bringing up secondary CPUs ... [ 0.059628] x86: Booting SMP configuration: [ 0.060031] .... node #0, CPUs: #1 #2 #3 [ 0.064037] smp: Brought up 1 node, 4 CPUs [ 0.066015] smpboot: Max logical packages: 1 [ 0.067017] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.159022] node 0 deferred pages initialised in 89ms [ 0.163061] devtmpfs: initialized [ 0.164266] x86/mm: Memory block size: 128MB [ 0.166736] gcov: version magic: 0x41383552 [ 0.170321] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.174143] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.176314] pinctrl core: initialized pinctrl subsystem [ 0.179195] [ 0.179789] ************************************************************* [ 0.182014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.185015] ** ** [ 0.187013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.189016] ** ** [ 0.191012] ** This means that this kernel is built to expose internal ** [ 0.193013] ** IOMMU data structures, which may compromise security on ** [ 0.195010] ** your system. ** [ 0.196020] ** ** [ 0.198009] ** If you see this message and you are not debugging the ** [ 0.200009] ** kernel, report this immediately to your vendor! ** [ 0.201016] ** ** [ 0.203010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.205012] ************************************************************* [ 0.207655] NET: Registered protocol family 16 [ 0.209385] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.212060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.215057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.218516] cpuidle: using governor menu [ 0.219270] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.222541] PCI: Using configuration type 1 for base access [ 0.225129] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.236057] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.238023] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.241246] cryptd: max_cpu_qlen set to 1000 [ 0.245344] ACPI: Added _OSI(Module Device) [ 0.247017] ACPI: Added _OSI(Processor Device) [ 0.249014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.250012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.255394] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.262590] ACPI: Interpreter enabled [ 0.263065] ACPI: PM: (supports S0 S3 S4 S5) [ 0.265012] ACPI: Using IOAPIC for interrupt routing [ 0.266075] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.270437] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.283159] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.286057] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.289023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.293104] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.299615] acpiphp: Slot [2] registered [ 0.301160] acpiphp: Slot [5] registered [ 0.303154] acpiphp: Slot [6] registered [ 0.305154] acpiphp: Slot [7] registered [ 0.307161] acpiphp: Slot [8] registered [ 0.308176] acpiphp: Slot [9] registered [ 0.310100] acpiphp: Slot [10] registered [ 0.311136] acpiphp: Slot [3] registered [ 0.313109] acpiphp: Slot [4] registered [ 0.314122] acpiphp: Slot [11] registered [ 0.316124] acpiphp: Slot [12] registered [ 0.318104] acpiphp: Slot [13] registered [ 0.319109] acpiphp: Slot [14] registered [ 0.321134] acpiphp: Slot [15] registered [ 0.322101] acpiphp: Slot [16] registered [ 0.324089] acpiphp: Slot [17] registered [ 0.325102] acpiphp: Slot [18] registered [ 0.327092] acpiphp: Slot [19] registered [ 0.328097] acpiphp: Slot [20] registered [ 0.329065] acpiphp: Slot [21] registered [ 0.331126] acpiphp: Slot [22] registered [ 0.332099] acpiphp: Slot [23] registered [ 0.334098] acpiphp: Slot [24] registered [ 0.335184] acpiphp: Slot [25] registered [ 0.337124] acpiphp: Slot [26] registered [ 0.339103] acpiphp: Slot [27] registered [ 0.341127] acpiphp: Slot [28] registered [ 0.342144] acpiphp: Slot [29] registered [ 0.344089] acpiphp: Slot [30] registered [ 0.345120] acpiphp: Slot [31] registered [ 0.347055] PCI host bridge to bus 0000:00 [ 0.348016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.350021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.352022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.355036] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.357029] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.361035] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.363174] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.366075] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.370372] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.380815] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.385061] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.388024] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.390015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.393020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.397599] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.399838] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.403039] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.406943] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.412016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.426923] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.431013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.437405] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.453015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.464025] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.481022] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.491472] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.498014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.507016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.526013] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.537502] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.547015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.554021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.578018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.587826] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.595024] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.604018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.626016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.642237] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.649017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.656019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.672019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.681774] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.691029] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.701019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.717029] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.733135] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.734450] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.737259] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.739372] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.741190] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.747137] iommu: Default domain type: Passthrough [ 0.749508] SCSI subsystem initialized [ 0.751149] ACPI: bus type USB registered [ 0.753094] usbcore: registered new interface driver usbfs [ 0.755090] usbcore: registered new interface driver hub [ 0.757107] usbcore: registered new device driver usb [ 0.759263] pps_core: LinuxPPS API ver. 1 registered [ 0.761013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.765072] PTP clock support registered [ 0.767109] EDAC MC: Ver: 3.0.0 [ 0.769235] PCI: Using ACPI for IRQ routing [ 0.771556] NetLabel: Initializing [ 0.773016] NetLabel: domain hash size = 128 [ 0.775013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.778085] NetLabel: unlabeled traffic allowed by default [ 0.780153] vgaarb: loaded [ 0.782212] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.783010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.789000] clocksource: Switched to clocksource kvm-clock [ 0.894716] VFS: Disk quotas dquot_6.6.0 [ 0.896881] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.899870] *** VALIDATE ramfs *** [ 0.901320] *** VALIDATE hugetlbfs *** [ 0.903577] pnp: PnP ACPI init [ 0.906234] pnp: PnP ACPI: found 6 devices [ 0.923823] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.927801] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.929688] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.931768] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.935297] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.937837] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.939738] NET: Registered protocol family 2 [ 0.941797] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.948033] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.951760] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.956406] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.961421] TCP: Hash tables configured (established 65536 bind 65536) [ 0.964503] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.967814] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.971564] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.975368] NET: Registered protocol family 1 [ 0.979199] RPC: Registered named UNIX socket transport module. [ 0.981675] RPC: Registered udp transport module. [ 0.984744] RPC: Registered tcp transport module. [ 0.987494] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.990278] NET: Registered protocol family 44 [ 0.992079] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.994413] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.996668] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.998525] PCI: CLS 0 bytes, default 64 [ 0.999713] Unpacking initramfs... [ 2.388371] debug: unmapping init [mem 0xffff9acb7cc54000-0xffff9acb7ffbffff] [ 2.391380] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.393805] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.399765] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 2.926930] Initialise system trusted keyrings [ 2.928765] Key type blacklist registered [ 2.931083] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.940785] zbud: loaded [ 2.944096] *** VALIDATE nfs *** [ 2.945545] *** VALIDATE nfs4 *** [ 2.947323] pstore: using deflate compression [ 2.950679] Platform Keyring initialized [ 3.049208] NET: Registered protocol family 38 [ 3.051746] Key type asymmetric registered [ 3.053243] Asymmetric key parser 'x509' registered [ 3.054977] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.059070] io scheduler mq-deadline registered [ 3.060790] io scheduler kyber registered [ 3.062459] io scheduler bfq registered [ 3.065057] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.068202] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.071792] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.075610] ACPI: Power Button [PWRF] [ 3.082314] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.090165] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.106433] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.115199] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.136907] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.164794] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.192796] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.198382] Non-volatile memory driver v1.3 [ 3.200411] Linux agpgart interface v0.103 [ 3.233920] virtio_blk virtio1: [vda] 145160 512-byte logical blocks (74.3 MB/70.9 MiB) [ 3.236610] vda: detected capacity change from 0 to 74321920 [ 3.255362] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.258654] vdb: detected capacity change from 0 to 1073741824 [ 3.277043] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.280074] vdc: detected capacity change from 0 to 2621440000 [ 3.294837] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.297765] vdd: detected capacity change from 0 to 2621440000 [ 3.320265] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.322786] vde: detected capacity change from 0 to 4294967296 [ 3.348121] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.351389] vdf: detected capacity change from 0 to 4294967296 [ 3.358628] libphy: Fixed MDIO Bus: probed [ 3.364236] usbcore: registered new interface driver usbserial_generic [ 3.366715] usbserial: USB Serial support registered for generic [ 3.369080] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.372655] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.374400] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.376782] mousedev: PS/2 mouse device common for all mice [ 3.379929] rtc_cmos 00:05: RTC can wake from S4 [ 3.382226] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.384800] rtc_cmos 00:05: registered as rtc0 [ 3.388364] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.391237] intel_pstate: CPU model not supported [ 3.393947] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.394973] hid: raw HID events driver (C) Jiri Kosina [ 3.399830] usbcore: registered new interface driver usbhid [ 3.401783] usbhid: USB HID core driver [ 3.403244] drop_monitor: Initializing network drop monitor service [ 3.403250] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.405372] Initializing XFRM netlink socket [ 3.410046] NET: Registered protocol family 10 [ 3.412528] Segment Routing with IPv6 [ 3.414093] NET: Registered protocol family 17 [ 3.415919] mpls_gso: MPLS GSO support [ 3.422908] RAS: Correctable Errors collector initialized. [ 3.425043] AVX version of gcm_enc/dec engaged. [ 3.426698] AES CTR mode by8 optimization enabled [ 3.504597] sched_clock: Marking stable (3504576711, 0)->(4426751506, -922174795) [ 3.508098] registered taskstats version 1 [ 3.509901] Loading compiled-in X.509 certificates [ 3.512030] zswap: loaded using pool lzo/zbud [ 3.542693] Key type big_key registered [ 3.555162] Key type encrypted registered [ 3.556939] ima: No TPM chip found, activating TPM-bypass! [ 3.558797] ima: Allocated hash algorithm: sha1 [ 3.560519] ima: No architecture policies found [ 3.562336] evm: Initialising EVM extended attributes: [ 3.564250] evm: security.selinux [ 3.565393] evm: security.ima [ 3.566542] evm: security.capability [ 3.567784] evm: HMAC attrs: 0x1 [ 3.570401] rtc_cmos 00:05: setting system clock to 2026-06-23 12:25:05 UTC (1782217505) [ 3.576497] debug: unmapping init [mem 0xffffffffa3a03000-0xffffffffa3bfffff] [ 3.581055] debug: unmapping init [mem 0xffffffffa2782000-0xffffffffa2a58fff] [ 3.593031] Write protecting the kernel read-only data: 28672k [ 3.596560] debug: unmapping init [mem 0xffffffffa0e03000-0xffffffffa0ffffff] [ 3.599511] debug: unmapping init [mem 0xffffffffa1714000-0xffffffffa17fffff] [ 3.634234] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.642861] systemd[1]: Detected virtualization kvm. [ 3.645923] systemd[1]: Detected architecture x86-64. [ 3.647893] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.677291] systemd[1]: No hostname configured. [ 3.678757] systemd[1]: Set hostname to . [ 3.680580] random: systemd: uninitialized urandom read (16 bytes read) [ 3.682331] systemd[1]: Initializing machine ID from random generator. [ 3.721906] random: ln: uninitialized urandom read (6 bytes read) [ 3.802080] random: systemd: uninitialized urandom read (16 bytes read) [ 3.805124] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.813982] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.818212] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... [ OK ] Reached target Swap. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.527940] device-mapper: uevent: version 1.0.3 [ 4.530739] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.521876] virtio_net virtio0 ens2: renamed from eth0 Starting dracut initqueue hook... [ 5.549019] random: fast init done [ 5.704982] scsi host0: ata_piix [ 5.722584] scsi host1: ata_piix [ 5.725601] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.728368] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.565964] dracut-initqueue[580]: RTNETLINK answers: File exists [ 10.073528] random: crng init done [ 10.075199] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.886159] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.017271] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.284578] SELinux: Disabled at runtime. [ 12.348592] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.357122] systemd[1]: Detected virtualization kvm. [ 12.359104] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.909306] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.913938] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.919987] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.925164] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.932429] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.939970] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.953628] systemd[1]: Listening on RPCbind Server Activation Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Kernel Debug File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Mounting Huge Pages File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-getty.slice. Starting Apply Kernel Variables... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target Paths. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ 13.097514] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting POSIX Message Queue File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 13.432545] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.759167] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.775609] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.923563] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.932980] EDAC sbridge: Ver: 1.1.2 [ 15.637125] Key type dns_resolver registered [ 15.968418] NFS: Registering the id_resolver key type [ 15.970836] Key type id_resolver registered [ 15.971845] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg416-server login: [ 70.496860] libcfs: loading out-of-tree module taints kernel. [ 70.554323] Key type ._llcrypt registered [ 70.557814] Key type .llcrypt registered [ 70.673832] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_hostid [ 88.754981] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 90.502144] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 90.524075] alg: No test for adler32 (adler32-zlib) [ 92.166761] Lustre: Lustre: Build Version: 2.17.54_83_g7183f09 [ 93.299511] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 95.143202] Key type lgssc registered [ 97.374579] Lustre: Echo OBD driver; http://www.lustre.org/ [ 105.224351] hrtimer: interrupt took 5340817 ns [ 116.733396] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 162.828993] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 176.527737] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 176.565713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 177.858743] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 177.920848] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 178.087413] Lustre: lustre-MDT0000: new disk, initializing [ 178.228434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 178.263924] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 182.574625] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 196.807609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 196.939283] Lustre: 6513:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 196.985063] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 196.990674] Lustre: Skipped 1 previous similar message [ 197.168254] Lustre: lustre-MDT0001: new disk, initializing [ 197.236539] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 197.263534] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 197.280536] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 201.351903] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 206.365771] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 215.667259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 215.952707] Lustre: lustre-OST0000: new disk, initializing [ 215.964581] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 215.973863] Lustre: 8451:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 216.064259] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 218.939106] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 218.947143] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 219.046700] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 222.819785] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 237.456765] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 237.632341] Lustre: lustre-OST0001: new disk, initializing [ 237.637588] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 237.645158] Lustre: 9524:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 237.708854] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 244.171590] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 245.282769] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 245.289574] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 245.387332] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 256.073672] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 261.964682] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 268.738561] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing check_logdir /tmp/testlogs/ [ 274.187300] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing yml_node [ 278.373784] Lustre: DEBUG MARKER: Client: 2.17.54.83 [ 280.942255] Lustre: DEBUG MARKER: MDS: 2.17.54.83 [ 283.585378] Lustre: DEBUG MARKER: OSS: 2.17.54.83 [ 285.342846] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Jun 23 08:29:45 EDT 2026 [ 303.014930] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 304.961566] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 320.480529] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 320.498686] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 320.512549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 322.016828] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 322.019477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 322.030158] Lustre: Skipped 2 previous similar messages [ 322.034489] Lustre: Skipped 2 previous similar messages [ 326.841455] Lustre: server umount lustre-MDT0000 complete [ 330.713216] LustreError: 6505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782217832 with bad export cookie 6118697465521318642 [ 330.720640] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 330.723361] LustreError: 6505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 332.256609] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 332.259348] LustreError: 8459:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 332.259513] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 332.259517] Lustre: Skipped 1 previous similar message [ 332.270810] Lustre: Skipped 1 previous similar message [ 332.283437] LustreError: 8459:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 335.296665] LustreError: 8459:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 335.298821] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 335.320826] LustreError: 8459:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 335.353984] Lustre: Skipped 2 previous similar messages [ 340.450164] LustreError: 12464:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 340.453163] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 340.483752] LustreError: 12464:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 340.509907] Lustre: Skipped 1 previous similar message [ 345.055114] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 345.221991] Lustre: server umount lustre-MDT0001 complete [ 349.629542] Lustre: server umount lustre-OST0000 complete [ 353.419453] Lustre: server umount lustre-OST0001 complete [ 360.958768] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_hostid [ 371.883977] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 408.796438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 409.179866] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 409.224810] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 409.348592] Lustre: lustre-MDT0000: new disk, initializing [ 409.488799] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 409.528216] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 413.742256] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 424.071902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 424.192887] Lustre: 15079:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 424.250716] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 424.253269] Lustre: Skipped 1 previous similar message [ 424.330337] Lustre: lustre-MDT0001: new disk, initializing [ 424.416116] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 424.453821] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 424.465969] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 430.473141] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 444.170982] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 449.093990] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 455.789289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 456.157820] Lustre: lustre-OST0000: new disk, initializing [ 456.163430] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 456.171751] Lustre: 17046:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 456.266924] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 457.995157] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 458.005712] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 458.178284] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 462.326966] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 471.622916] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 476.245250] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 476.459964] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 481.277818] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 481.511134] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 483.811773] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 483.822184] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 483.823928] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 483.824026] Lustre: Skipped 1 previous similar message [ 483.848882] LustreError: Skipped 1 previous similar message [ 489.438474] Lustre: server umount lustre-OST0000 complete [ 494.053156] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 494.062667] Lustre: Skipped 1 previous similar message [ 499.534747] Lustre: server umount lustre-MDT0000 complete [ 503.424956] LustreError: 15071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782218005 with bad export cookie 6118697465521320175 [ 503.436855] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 503.757446] Lustre: server umount lustre-MDT0001 complete [ 507.092494] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 08:33:27 (1782218007) [ 513.219636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 513.692135] LustreError: 19312:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 513.755538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 518.384273] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 519.139637] LustreError: 19313:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 526.634801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 531.458207] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 539.262095] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 545.149669] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 552.358422] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 560.280399] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 568.513863] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 581.091232] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 581.099660] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 581.134352] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 581.139896] Lustre: Skipped 7 previous similar messages [ 585.965713] Lustre: server umount lustre-OST0000 complete [ 596.713659] Lustre: server umount lustre-MDT0000 complete [ 596.964340] LustreError: 21399:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 596.977978] LustreError: 21399:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 600.432736] LustreError: 19292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782218102 with bad export cookie 6118697465521321057 [ 600.445694] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 600.448727] LustreError: 19292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 600.982401] Lustre: server umount lustre-MDT0001 complete [ 613.468423] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 616.835381] Key type lgssc unregistered [ 617.261779] LNet: 22845:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 617.273303] LNetError: 22845:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 617.294321] LNet: Removed LNI 192.168.204.116@tcp [ 618.359332] Key type .llcrypt unregistered [ 618.364350] Key type ._llcrypt unregistered [ 643.545569] Key type ._llcrypt registered [ 643.547942] Key type .llcrypt registered [ 643.687411] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 645.099947] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 645.209947] alg: No test for adler32 (adler32-zlib) [ 646.283607] Lustre: Lustre: Build Version: 2.17.54_83_g7183f09 [ 646.638942] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 648.415136] Key type lgssc registered [ 649.802971] Lustre: Echo OBD driver; http://www.lustre.org/ [ 658.829690] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 658.872494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 660.310602] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 664.719390] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 670.688792] LustreError: 24257:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 670.714658] LustreError: 24257:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 672.944685] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 673.238805] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 679.477794] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 686.689251] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 692.832947] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 699.214185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 699.640928] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 705.508820] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 705.774367] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 712.769485] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 729.569214] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 729.577276] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 729.586096] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 729.593549] Lustre: Skipped 1 previous similar message [ 729.605818] Lustre: Skipped 1 previous similar message [ 731.356612] Lustre: server umount lustre-OST0000 complete [ 739.808298] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 739.820202] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 739.826860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 739.836442] Lustre: Skipped 1 previous similar message [ 741.299528] Lustre: server umount lustre-MDT0000 complete [ 744.839277] LustreError: 24236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782218246 with bad export cookie 9530692783202224479 [ 744.849294] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 744.861563] LustreError: 24236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 745.234699] Lustre: server umount lustre-MDT0001 complete [ 756.083140] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 759.001924] Key type lgssc unregistered [ 759.434757] LNet: 28235:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 759.442325] LNetError: 28235:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 759.464928] LNet: Removed LNI 192.168.204.116@tcp [ 760.387174] Key type .llcrypt unregistered [ 760.392180] Key type ._llcrypt unregistered [ 784.897797] Key type ._llcrypt registered [ 784.901889] Key type .llcrypt registered [ 785.033350] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 786.073654] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 786.244222] alg: No test for adler32 (adler32-zlib) [ 787.344296] Lustre: Lustre: Build Version: 2.17.54_83_g7183f09 [ 787.596173] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 789.271177] Key type lgssc registered [ 790.430961] Lustre: Echo OBD driver; http://www.lustre.org/ [ 798.387990] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 798.412447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 799.832021] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 804.270703] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 809.967542] LustreError: 29649:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 809.982578] LustreError: 29649:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 812.298594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 812.696493] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 817.252331] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 825.335087] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 831.583913] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 844.821206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 845.199654] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 851.791490] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 859.402557] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 869.816675] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 878.048398] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 878.063232] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 878.080401] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 880.098548] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 880.109545] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 880.118064] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 880.153298] Lustre: Skipped 1 previous similar message [ 885.229068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 885.240897] Lustre: Skipped 1 previous similar message [ 890.336667] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 890.342071] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 890.382376] Lustre: Skipped 1 previous similar message [ 890.600300] Lustre: server umount lustre-OST0000 complete [ 895.456845] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 895.469442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 895.475670] Lustre: Skipped 1 previous similar message [ 905.698205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 905.704870] Lustre: Skipped 2 previous similar messages [ 909.279246] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 910.817360] LustreError: 29650:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 910.844293] LustreError: 29650:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 911.523585] Lustre: server umount lustre-MDT0000 complete [ 915.687291] LustreError: 29629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782218417 with bad export cookie 12713292504506072376 [ 915.699798] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 915.704921] LustreError: 29629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 915.981418] Lustre: server umount lustre-MDT0001 complete [ 928.267750] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 931.602756] Key type lgssc unregistered [ 931.947883] LNet: 33553:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 931.959222] LNetError: 33553:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 931.976549] LNet: Removed LNI 192.168.204.116@tcp [ 932.850512] Key type .llcrypt unregistered [ 932.852362] Key type ._llcrypt unregistered [ 957.150049] Key type ._llcrypt registered [ 957.160525] Key type .llcrypt registered [ 957.307652] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 958.532338] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 958.771799] alg: No test for adler32 (adler32-zlib) [ 959.834696] Lustre: Lustre: Build Version: 2.17.54_83_g7183f09 [ 960.161228] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 961.855184] Key type lgssc registered [ 963.346759] Lustre: Echo OBD driver; http://www.lustre.org/ [ 971.761401] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 971.783470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 973.451608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 977.997756] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 984.035670] LustreError: 34966:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 984.069796] LustreError: 34966:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 986.890181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 987.260519] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 991.991949] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 999.660679] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1005.602751] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1013.106086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1013.472442] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1018.617253] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1018.912909] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 1025.405041] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1038.307857] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1038.317622] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1038.322176] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1038.333797] Lustre: Skipped 1 previous similar message [ 1039.640228] Lustre: server umount lustre-OST0000 complete [ 1043.426231] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1043.428088] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1043.446801] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1043.462083] Lustre: Skipped 2 previous similar messages [ 1048.546226] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1048.560043] Lustre: Skipped 1 previous similar message [ 1048.836432] Lustre: server umount lustre-MDT0000 complete [ 1052.264406] LustreError: 34947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782218554 with bad export cookie 6235392320536320223 [ 1052.275518] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1052.279581] LustreError: 34947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1052.598547] Lustre: server umount lustre-MDT0001 complete [ 1061.749155] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 1064.166634] Key type lgssc unregistered [ 1064.477672] LNet: 38519:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1064.487248] LNetError: 38519:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1065.512211] LNet: Removed LNI 192.168.204.116@tcp [ 1066.437943] Key type .llcrypt unregistered [ 1066.442656] Key type ._llcrypt unregistered [ 1089.285023] Key type ._llcrypt registered [ 1089.287274] Key type .llcrypt registered [ 1089.391109] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 1090.181585] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1090.211924] alg: No test for adler32 (adler32-zlib) [ 1091.234268] Lustre: Lustre: Build Version: 2.17.54_83_g7183f09 [ 1091.470356] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 1093.151169] Key type lgssc registered [ 1094.390777] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1101.289891] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1101.306969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1102.912083] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1107.291464] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 1113.058735] LustreError: 39941:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1113.073683] LustreError: 39941:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1115.394717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1115.852827] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1120.117579] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 1127.362695] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1132.633800] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1139.548188] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1140.068810] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1146.381890] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 1147.253318] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1153.274464] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1159.751757] Lustre: lustre-OST0000: Not available for connect from 192.168.204.16@tcp (stopping) [ 1161.701339] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1161.708745] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1161.729789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1162.729794] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1162.730348] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1162.736510] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1162.752179] Lustre: Skipped 1 previous similar message [ 1164.892189] Lustre: server umount lustre-OST0000 complete [ 1187.235970] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1187.238534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1187.248023] Lustre: Skipped 1 previous similar message [ 1190.516959] Lustre: server umount lustre-MDT0000 complete [ 1192.419679] LustreError: 40987:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1192.436459] LustreError: 40987:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1193.903989] LustreError: 39921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782218695 with bad export cookie 7836083914480740340 [ 1193.907059] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1193.914178] LustreError: 39921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1208.287195] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1208.565227] Lustre: server umount lustre-MDT0001 complete [ 1219.047164] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 1221.919413] Key type lgssc unregistered [ 1222.219534] LNet: 43649:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1222.225681] LNetError: 43649:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1222.241321] LNet: Removed LNI 192.168.204.116@tcp [ 1223.116624] Key type .llcrypt unregistered [ 1223.121128] Key type ._llcrypt unregistered [ 1246.441757] Key type ._llcrypt registered [ 1246.443996] Key type .llcrypt registered [ 1246.575874] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 1248.053555] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1248.132152] alg: No test for adler32 (adler32-zlib) [ 1249.288600] Lustre: Lustre: Build Version: 2.17.54_83_g7183f09 [ 1249.609116] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 1251.256839] Key type lgssc registered [ 1252.403361] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1259.522053] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1259.539906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1260.961714] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1265.402852] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 1270.243560] LustreError: 45061:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1270.267889] LustreError: 45061:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1273.824632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1274.219426] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1279.055441] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 1286.863344] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1291.796756] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1298.248670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1298.670200] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1305.213517] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 1305.839293] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1312.145399] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1320.016953] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.16@tcp (stopping) [ 1320.419676] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1320.433701] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1321.439994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1321.440798] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1321.456702] Lustre: Skipped 1 previous similar message [ 1325.087172] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.16@tcp (stopping) [ 1325.102664] Lustre: Skipped 2 previous similar messages [ 1325.613974] Lustre: server umount lustre-MDT0000 complete [ 1326.564239] LustreError: 45062:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1326.589951] LustreError: 45062:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1330.046510] LustreError: 45040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782218831 with bad export cookie 13047737239121478810 [ 1330.062179] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1330.209381] LustreError: 46302:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1330.275287] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.16@tcp (stopping) [ 1331.692779] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1331.720412] Lustre: Skipped 1 previous similar message [ 1335.361865] LustreError: 45057:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1335.362071] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.16@tcp (stopping) [ 1335.385594] LustreError: 45057:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1335.417026] Lustre: Skipped 1 previous similar message [ 1337.012703] Lustre: server umount lustre-MDT0001 complete [ 1359.020838] Lustre: server umount lustre-OST0000 complete [ 1377.216592] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 1381.103467] Key type lgssc unregistered [ 1381.384083] LNet: 48924:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1381.392630] LNetError: 48924:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1381.417726] LNet: Removed LNI 192.168.204.116@tcp [ 1382.506172] Key type .llcrypt unregistered [ 1382.508576] Key type ._llcrypt unregistered [ 1408.918365] Key type ._llcrypt registered [ 1408.922130] Key type .llcrypt registered [ 1409.076402] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 1410.375847] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1410.454440] alg: No test for adler32 (adler32-zlib) [ 1411.562025] Lustre: Lustre: Build Version: 2.17.54_83_g7183f09 [ 1411.896345] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 1413.655215] Key type lgssc registered [ 1414.936707] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1422.782238] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1422.802489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1438.367422] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1443.487808] LustreError: 50314:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.116@tcp: failed processing log, type 4: rc = -110 [ 1470.177452] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1476.208712] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 1482.775505] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1649.732953] Lustre: server umount lustre-OST0000 complete [ 1665.853942] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 1668.431600] Key type lgssc unregistered [ 1668.652306] LNet: 51989:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1668.660738] LNetError: 51989:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1668.671894] LNet: Removed LNI 192.168.204.116@tcp [ 1669.365523] Key type .llcrypt unregistered [ 1669.371388] Key type ._llcrypt unregistered [ 1691.845337] Key type ._llcrypt registered [ 1691.848847] Key type .llcrypt registered [ 1691.976395] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 1692.841740] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1692.964350] alg: No test for adler32 (adler32-zlib) [ 1694.032913] Lustre: Lustre: Build Version: 2.17.54_83_g7183f09 [ 1694.293329] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 1695.967331] Key type lgssc registered [ 1697.038279] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1704.787228] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1704.817833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1706.297429] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1710.738325] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 1716.719515] LustreError: 53403:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1716.741958] LustreError: 53403:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1718.284044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1718.704942] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1723.235353] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 1730.496783] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1735.970475] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1741.810049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1742.160686] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1747.190148] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 1751.435854] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1755.367289] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1758.690264] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1758.694535] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1758.704252] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1761.773201] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1761.782836] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1761.784036] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1764.295767] Lustre: server umount lustre-OST0000 complete [ 1769.953223] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1769.960983] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1769.975502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1769.979796] Lustre: Skipped 1 previous similar message [ 1772.000429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1772.003468] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1775.357816] Lustre: server umount lustre-MDT0000 complete [ 1777.121283] LustreError: 54130:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1777.144457] LustreError: 54130:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1778.711784] LustreError: 53383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782219280 with bad export cookie 1695081655972254814 [ 1778.732062] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1778.985353] Lustre: server umount lustre-MDT0001 complete [ 1789.233682] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 1791.908347] Key type lgssc unregistered [ 1792.268682] LNet: 56934:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1792.273323] LNetError: 56934:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1793.328483] LNet: Removed LNI 192.168.204.116@tcp [ 1794.519752] Key type .llcrypt unregistered [ 1794.521517] Key type ._llcrypt unregistered [ 1818.021047] Key type ._llcrypt registered [ 1818.022824] Key type .llcrypt registered [ 1818.145747] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 1819.391353] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1819.416517] alg: No test for adler32 (adler32-zlib) [ 1820.563857] Lustre: Lustre: Build Version: 2.17.54_83_g7183f09 [ 1820.837342] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 1822.503146] Key type lgssc registered [ 1823.644516] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1830.370674] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1830.379709] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1845.919453] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1851.039585] LustreError: 58321:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.116@tcp: failed processing log, type 4: rc = -110 [ 1877.791221] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1884.049751] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 1891.475301] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1899.383286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1899.876559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1899.963253] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1904.791669] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 1905.126514] LustreError: 58347:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1905.145263] LustreError: 58347:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1910.245330] LustreError: 58346:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1913.104559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1913.332525] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1913.353898] Lustre: Skipped 1 previous similar message [ 1913.558228] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1919.364483] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 1927.509341] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1933.655840] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1939.428049] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1939.441767] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1939.467896] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1942.305580] Lustre: server umount lustre-OST0000 complete [ 1964.513420] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1964.517647] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1964.550619] Lustre: Skipped 1 previous similar message [ 1964.559182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1964.568521] Lustre: Skipped 1 previous similar message [ 1970.146841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1970.152065] Lustre: Skipped 3 previous similar messages [ 1975.269479] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1975.279651] Lustre: Skipped 1 previous similar message [ 1978.337862] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1978.575728] Lustre: server umount lustre-MDT0000 complete [ 1979.362275] LustreError: 59363:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1979.390520] LustreError: 59363:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1982.905222] LustreError: 58327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782219484 with bad export cookie 14855084063476288844 [ 1982.931568] LustreError: 58327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1982.944294] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1997.280472] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1997.909943] Lustre: server umount lustre-MDT0001 complete [ 2010.294121] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 2013.861543] Key type lgssc unregistered [ 2014.268710] LNet: 62071:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2014.274478] LNetError: 62071:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2015.335759] LNet: Removed LNI 192.168.204.116@tcp [ 2016.422586] Key type .llcrypt unregistered [ 2016.424042] Key type ._llcrypt unregistered [ 2039.710412] Key type ._llcrypt registered [ 2039.714710] Key type .llcrypt registered [ 2039.837070] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 2040.978075] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2041.022685] alg: No test for adler32 (adler32-zlib) [ 2042.084818] Lustre: Lustre: Build Version: 2.17.54_83_g7183f09 [ 2042.291863] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 2043.951276] Key type lgssc registered [ 2044.948777] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2052.388686] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2052.420629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2053.994882] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2058.947337] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2063.337123] LustreError: 63487:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2063.354657] LustreError: 63487:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2067.628530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2067.910063] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2072.821575] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2081.223657] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2087.433178] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2094.692423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2095.253220] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2098.368352] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2102.658984] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2112.445319] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2129.382574] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2129.409279] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2129.420995] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2130.778533] Lustre: server umount lustre-OST0000 complete [ 2134.496770] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2134.501879] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2134.514030] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2138.593682] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2138.595033] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2143.724879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2143.727068] Lustre: Skipped 1 previous similar message [ 2148.320056] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2148.520811] Lustre: server umount lustre-MDT0000 complete [ 2148.833139] LustreError: 63487:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2148.843753] LustreError: 63487:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2149.855163] Lustre: 62797:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782219635/real 1782219635] req@00000000c1f566eb x1868792640600576/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1782219651 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2149.860413] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2149.890362] Lustre: 62797:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2149.928390] LustreError: 63488:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2149.956566] LustreError: 63488:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2152.648748] LustreError: 63467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782219654 with bad export cookie 2155813418457685273 [ 2152.659419] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2153.064502] Lustre: server umount lustre-MDT0001 complete [ 2168.729802] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 2172.300783] Key type lgssc unregistered [ 2172.548647] LNet: 67057:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2172.556970] LNetError: 67057:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2172.569890] LNet: Removed LNI 192.168.204.116@tcp [ 2173.296318] Key type .llcrypt unregistered [ 2173.298793] Key type ._llcrypt unregistered [ 2198.775364] Key type ._llcrypt registered [ 2198.785925] Key type .llcrypt registered [ 2198.945559] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 2199.860084] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2199.895812] alg: No test for adler32 (adler32-zlib) [ 2201.027306] Lustre: Lustre: Build Version: 2.17.54_83_g7183f09 [ 2201.298573] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 2203.031203] Key type lgssc registered [ 2204.435889] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2214.672198] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 09:01:54 (1782219714) [ 2222.233643] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2222.269457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2223.874549] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2229.574241] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2233.324258] LustreError: 68759:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2233.357359] LustreError: 68759:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2238.434568] LustreError: 68758:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2238.672644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2239.126077] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2244.244171] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2252.433498] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2258.692266] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2265.618577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2265.974637] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2269.059701] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2272.631780] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2280.111968] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2290.366656] Lustre: Failing over lustre-MDT0000 [ 2290.638696] Lustre: server umount lustre-MDT0000 complete [ 2293.323882] LustreError: 71019:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2293.737979] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2293.776723] Lustre: Skipped 2 previous similar messages [ 2298.397748] LustreError: 71019:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2298.421415] LustreError: 71019:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2301.079562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2301.202585] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2301.398508] Lustre: *** cfs_fail_loc=135, val=0*** [ 2301.504249] LustreError: 71848:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2301.509458] LustreError: 71848:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.116@tcp: cfg command failed: rc = -2 [ 2301.517228] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2301.517228] [ 2301.526501] LustreError: MGC192.168.204.116@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2301.546951] LustreError: 71836:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2301.553645] LustreError: 71836:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2301.567949] LustreError: 71836:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2301.577660] LustreError: 71836:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2301.603075] Lustre: server umount lustre-MDT0000 complete [ 2301.607173] LustreError: 71836:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2303.514617] LustreError: 68755:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2303.541688] LustreError: 68755:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2309.602800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2309.769732] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2309.780465] LustreError: 68752:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2310.165812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2310.224576] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2313.754550] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2315.242804] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2315.271253] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 2315.324421] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2315.476750] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2324.452487] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2324.465141] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2324.492472] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2325.487952] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2325.490888] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2328.295316] Lustre: server umount lustre-OST0000 complete [ 2335.714590] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2335.724408] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2335.742470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2335.759779] Lustre: Skipped 2 previous similar messages [ 2338.752203] Lustre: server umount lustre-MDT0000 complete [ 2340.834792] LustreError: 68755:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2340.851933] LustreError: 68755:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 2342.478311] LustreError: 68739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782219844 with bad export cookie 7218097009759925953 [ 2342.481220] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2342.492879] LustreError: 68739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2342.809477] Lustre: server umount lustre-MDT0001 complete [ 2354.603286] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 2357.390619] Key type lgssc unregistered [ 2357.642822] LNet: 73627:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2357.648594] LNetError: 73627:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2357.662860] LNet: Removed LNI 192.168.204.116@tcp [ 2358.512957] Key type .llcrypt unregistered [ 2358.519456] Key type ._llcrypt unregistered [ 2386.863746] Key type ._llcrypt registered [ 2386.867854] Key type .llcrypt registered [ 2386.960392] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 2388.392967] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2388.415735] alg: No test for adler32 (adler32-zlib) [ 2389.596412] Lustre: Lustre: Build Version: 2.17.54_83_g7183f09 [ 2389.911849] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 2391.567350] Key type lgssc registered [ 2392.938421] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2401.356478] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2401.378556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2403.010532] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2407.978629] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2412.518176] LustreError: 75040:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2412.533934] LustreError: 75040:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2417.645105] LustreError: 75041:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2418.003924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2418.638750] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2424.221867] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2431.361476] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2437.067773] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2443.695443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2444.089102] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2447.151519] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2449.879302] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2459.186939] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2469.853434] Lustre: Failing over lustre-MDT0000 [ 2469.867942] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -19 [ 2469.893450] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2469.941324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2470.305702] Lustre: server umount lustre-MDT0000 complete [ 2472.500120] LustreError: 75037:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2473.963956] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2473.983752] Lustre: Skipped 1 previous similar message [ 2477.600691] LustreError: 75036:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2477.613241] LustreError: 75036:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2480.422961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2480.671225] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2480.888145] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2481.142185] LustreError: 78128:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2481.161827] LustreError: 78128:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.116@tcp: cfg command failed: rc = -9 [ 2481.194624] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2481.194624] [ 2481.214166] LustreError: MGC192.168.204.116@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2481.252658] LustreError: 78117:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2481.275899] LustreError: 78117:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 2481.295285] LustreError: 78117:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2481.308214] LustreError: 78117:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2481.365527] Lustre: server umount lustre-MDT0000 complete [ 2481.375454] LustreError: 78117:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2482.712337] LustreError: 75037:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2482.740930] LustreError: 75037:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2488.275499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2488.483914] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2488.503975] LustreError: 75032:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2488.857577] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2488.968160] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2492.958674] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2493.942687] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2493.980714] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2494.024648] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2494.269608] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2504.162163] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2504.182270] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2504.193435] Lustre: Skipped 1 previous similar message [ 2509.090627] Lustre: server umount lustre-OST0000 complete [ 2514.403861] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2514.404935] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2514.439238] Lustre: Skipped 1 previous similar message [ 2514.460967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2514.475345] Lustre: Skipped 1 previous similar message [ 2519.521718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2519.527975] Lustre: Skipped 1 previous similar message [ 2522.013473] Lustre: server umount lustre-MDT0000 complete [ 2524.642452] LustreError: 75769:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2524.666870] LustreError: 75769:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2526.215140] LustreError: 75776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782220028 with bad export cookie 15190096667041346793 [ 2526.220069] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2526.238614] LustreError: 75776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2526.542337] Lustre: server umount lustre-MDT0001 complete [ 2537.009461] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 2539.381478] Key type lgssc unregistered [ 2539.679929] LNet: 79911:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2539.690734] LNetError: 79911:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2539.709785] LNet: Removed LNI 192.168.204.116@tcp [ 2540.726349] Key type .llcrypt unregistered [ 2540.729283] Key type ._llcrypt unregistered [ 2553.629782] Key type ._llcrypt registered [ 2553.632047] Key type .llcrypt registered [ 2553.779554] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 2554.162875] Key type .llcrypt unregistered [ 2554.164878] Key type ._llcrypt unregistered [ 2578.358367] Key type ._llcrypt registered [ 2578.360597] Key type .llcrypt registered [ 2578.467062] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 2579.175400] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2579.280103] alg: No test for adler32 (adler32-zlib) [ 2580.376684] Lustre: Lustre: Build Version: 2.17.54_83_g7183f09 [ 2580.678391] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 2582.415348] Key type lgssc registered [ 2583.949603] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2593.802405] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2593.835349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2595.367704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2599.497937] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2605.537827] LustreError: 81813:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2605.549896] LustreError: 81813:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2608.673579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2609.377577] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2613.915939] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2622.208843] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2628.722226] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2636.396033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2636.851143] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2640.949684] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2642.819344] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2650.770232] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2664.423379] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2664.425363] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2664.454277] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2664.469618] Lustre: Skipped 1 previous similar message [ 2664.479564] Lustre: Skipped 1 previous similar message [ 2669.538412] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2669.552194] Lustre: Skipped 1 previous similar message [ 2670.379992] Lustre: server umount lustre-OST0000 complete [ 2674.661996] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2674.692560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2679.779022] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2679.787626] Lustre: Skipped 2 previous similar messages [ 2680.410273] Lustre: server umount lustre-MDT0000 complete [ 2684.221744] LustreError: 81796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782220186 with bad export cookie 8919837615245377339 [ 2684.235435] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2684.241348] LustreError: 81796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2684.637029] Lustre: server umount lustre-MDT0001 complete [ 2694.915670] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 2697.499636] Key type lgssc unregistered [ 2697.795943] LNet: 85369:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2697.803532] LNetError: 85369:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2697.821859] LNet: Removed LNI 192.168.204.116@tcp [ 2698.765342] Key type .llcrypt unregistered [ 2698.772481] Key type ._llcrypt unregistered [ 2723.075751] Key type ._llcrypt registered [ 2723.079391] Key type .llcrypt registered [ 2723.281622] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 2724.305425] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2724.434703] alg: No test for adler32 (adler32-zlib) [ 2725.818920] Lustre: Lustre: Build Version: 2.17.54_83_g7183f09 [ 2726.291440] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 2728.015317] Key type lgssc registered [ 2729.871899] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2739.547891] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2739.585527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2741.127098] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2745.883055] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2751.465538] LustreError: 86788:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2751.497363] LustreError: 86788:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2754.669871] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2755.220733] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2760.141322] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2767.929821] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2774.387683] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2782.077870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2782.595132] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2786.776604] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2790.552801] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2797.194750] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2806.240451] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2806.253834] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2806.266425] Lustre: Skipped 1 previous similar message [ 2806.267250] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2808.086755] Lustre: server umount lustre-OST0000 complete [ 2816.480205] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2816.480482] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2816.501066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2816.511717] Lustre: Skipped 1 previous similar message [ 2816.530972] Lustre: Skipped 2 previous similar messages [ 2817.871166] Lustre: server umount lustre-MDT0000 complete [ 2821.468239] LustreError: 86769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782220323 with bad export cookie 8988083720627205046 [ 2821.489943] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2821.865930] Lustre: server umount lustre-MDT0001 complete [ 2832.136250] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 2835.050242] Key type lgssc unregistered [ 2835.439062] LNet: 90318:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2835.454063] LNetError: 90318:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2835.468901] LNet: Removed LNI 192.168.204.116@tcp [ 2836.348979] Key type .llcrypt unregistered [ 2836.353311] Key type ._llcrypt unregistered [ 2858.414071] Key type ._llcrypt registered [ 2858.418349] Key type .llcrypt registered [ 2858.524905] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 2859.563096] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2859.692334] alg: No test for adler32 (adler32-zlib) [ 2860.799625] Lustre: Lustre: Build Version: 2.17.54_83_g7183f09 [ 2861.143974] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 2862.871221] Key type lgssc registered [ 2864.183320] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2872.544779] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2872.558341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2874.155188] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2878.190612] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2884.579405] LustreError: 91734:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2884.590708] LustreError: 91734:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2886.776107] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2887.343668] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2892.901581] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2901.208823] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2906.533188] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2912.914791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2913.233848] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2918.639319] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2919.415387] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2924.317649] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2945.001403] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2945.008492] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2945.036951] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2947.237284] Lustre: server umount lustre-OST0000 complete [ 2953.185499] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2953.201400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2953.209703] Lustre: Skipped 1 previous similar message [ 2957.096206] Lustre: server umount lustre-MDT0000 complete [ 2958.304657] LustreError: 94223:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2958.331910] LustreError: 94223:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2961.185112] LustreError: 91713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782220463 with bad export cookie 7247356198492914491 [ 2961.215823] LustreError: 91713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2961.217466] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2971.785542] Lustre: server umount lustre-MDT0001 complete [ 2982.159299] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 2984.537817] Key type lgssc unregistered [ 2984.852704] LNet: 95290:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2984.863443] LNetError: 95290:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2984.876689] LNet: Removed LNI 192.168.204.116@tcp [ 2985.617968] Key type .llcrypt unregistered [ 2985.619717] Key type ._llcrypt unregistered [ 3009.625850] Key type ._llcrypt registered [ 3009.627937] Key type .llcrypt registered [ 3009.700675] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 3010.695842] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3010.742039] alg: No test for adler32 (adler32-zlib) [ 3011.879729] Lustre: Lustre: Build Version: 2.17.54_83_g7183f09 [ 3012.153579] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 3013.831408] Key type lgssc registered [ 3015.578101] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3025.375402] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3025.396892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3040.991318] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3046.111422] LustreError: 96677:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.116@tcp: failed processing log, type 4: rc = -110 [ 3072.799271] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3078.375434] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 3085.848166] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3091.496559] Lustre: server umount lustre-OST0000 complete [ 3099.185946] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 09:16:39 (1782220599) [ 3105.048504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3105.717461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3110.294756] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 3110.883589] LustreError: 98318:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3110.899724] LustreError: 98318:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3116.015209] LustreError: 98319:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3118.838624] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3119.100677] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3124.072895] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 3131.356924] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3136.799467] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3144.078031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3144.435682] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3150.580090] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3150.835470] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 3158.680217] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3174.368659] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3174.373631] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3174.390463] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3174.406267] Lustre: Skipped 1 previous similar message [ 3179.500637] LustreError: 100139:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3179.520996] LustreError: 100139:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3179.589643] Lustre: server umount lustre-OST0000 complete [ 3184.608786] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3184.634650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3184.644997] Lustre: Skipped 2 previous similar messages [ 3189.550399] Lustre: server umount lustre-MDT0000 complete [ 3189.730744] LustreError: 98313:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3189.754044] LustreError: 98313:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3193.434306] LustreError: 98300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782220695 with bad export cookie 8083705557872813565 [ 3193.448726] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3193.983496] Lustre: server umount lustre-MDT0001 complete [ 3209.317937] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 3213.656630] Key type lgssc unregistered [ 3213.983047] LNet: 102008:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3214.004394] LNetError: 102008:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3214.043621] LNet: Removed LNI 192.168.204.116@tcp [ 3215.504865] Key type .llcrypt unregistered [ 3215.507842] Key type ._llcrypt unregistered [ 3256.215451] Key type ._llcrypt registered [ 3256.218141] Key type .llcrypt registered [ 3256.312251] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 3257.196602] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3257.230437] alg: No test for adler32 (adler32-zlib) [ 3258.266657] Lustre: Lustre: Build Version: 2.17.54_83_g7183f09 [ 3258.582220] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 3260.224536] Key type lgssc registered [ 3261.410379] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3268.884607] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3268.901670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3270.490512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3274.607315] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 3280.877778] LustreError: 104194:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3280.920834] LustreError: 104194:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3282.954948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3283.507030] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3287.881607] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 3295.202726] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3300.958755] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3307.646368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3307.982374] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3314.207567] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 3315.135278] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3320.821266] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3329.511777] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3329.515617] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3329.524435] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3330.532634] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3330.536306] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3330.544437] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3335.650825] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3335.654986] Lustre: Skipped 2 previous similar messages [ 3340.771524] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3340.784728] Lustre: Skipped 1 previous similar message [ 3343.327520] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3343.782868] Lustre: server umount lustre-OST0000 complete [ 3349.985632] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3349.991407] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3350.008975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3361.249192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3361.261082] Lustre: Skipped 5 previous similar messages [ 3362.271155] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3362.538635] Lustre: server umount lustre-MDT0000 complete [ 3366.367991] LustreError: 104921:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3366.394474] LustreError: 104921:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3366.965336] LustreError: 104173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782220868 with bad export cookie 4720544627363634291 [ 3366.984466] LustreError: 104173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3366.986100] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3367.580928] Lustre: server umount lustre-MDT0001 complete [ 3379.158432] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 3382.471182] Key type lgssc unregistered [ 3382.753685] LNet: 107739:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3382.762215] LNetError: 107739:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3382.775943] LNet: Removed LNI 192.168.204.116@tcp [ 3383.434410] Key type .llcrypt unregistered [ 3383.437405] Key type ._llcrypt unregistered [ 3403.136458] Key type ._llcrypt registered [ 3403.139702] Key type .llcrypt registered [ 3403.251158] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 3404.251890] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3404.312810] alg: No test for adler32 (adler32-zlib) [ 3405.349632] Lustre: Lustre: Build Version: 2.17.54_83_g7183f09 [ 3405.623046] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 3407.272628] Key type lgssc registered [ 3408.192720] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3416.106514] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3416.123438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3431.775773] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3436.896866] LustreError: 108983:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.116@tcp: failed processing log, type 4: rc = -110 [ 3463.583838] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3468.337261] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 3475.328228] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3482.873435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3483.174229] LustreError: MGC192.168.204.116@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3483.188886] LustreError: 110008:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3483.197675] LustreError: 110008:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3483.208928] LustreError: 110008:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3483.315355] Lustre: server umount lustre-MDT0000 complete [ 3483.320218] LustreError: 110008:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3504.607170] Lustre: 108320:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782220990/real 1782220990] req@00000000e250a4e0 x1868794069781760/t0(0) o400->MGC192.168.204.116@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1782221006 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3504.608692] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3504.650904] Lustre: 108320:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3504.661110] LustreError: 108995:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3504.811457] Lustre: server umount lustre-OST0000 complete [ 3514.055405] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_hostid [ 3523.477537] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 3558.588115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3558.838766] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3558.870894] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3558.951917] Lustre: lustre-MDT0000: new disk, initializing [ 3559.055205] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3559.077867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3563.068561] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 3572.797506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3572.933452] Lustre: 112798:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3572.977431] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3572.981369] Lustre: Skipped 1 previous similar message [ 3573.041787] Lustre: lustre-MDT0001: new disk, initializing [ 3573.143225] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3573.184097] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3573.197483] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3577.561292] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 3586.056988] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3591.097082] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3597.259993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3597.567298] Lustre: lustre-OST0000: new disk, initializing [ 3597.571310] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3597.579564] Lustre: 114766:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3597.675232] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3604.154976] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 3605.582605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3605.593269] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3605.722715] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3613.311356] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3617.729490] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3617.939267] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3622.599766] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3622.855290] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3626.466197] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3626.469544] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3626.502550] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3626.518911] LustreError: Skipped 1 previous similar message [ 3631.093577] Lustre: server umount lustre-OST0000 complete [ 3636.706978] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3636.730173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3636.730182] Lustre: Skipped 1 previous similar message [ 3636.749053] Lustre: Skipped 2 previous similar messages [ 3641.377448] Lustre: server umount lustre-MDT0000 complete [ 3645.027480] LustreError: 112790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782221146 with bad export cookie 11683936366230543926 [ 3645.031800] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3645.036427] LustreError: 112790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3645.387204] Lustre: server umount lustre-MDT0001 complete [ 3653.328839] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 09:25:53 (1782221153) [ 3656.484294] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3671.735201] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_hostid [ 3679.701153] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 3717.058363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3717.246627] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3717.264746] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3717.316374] Lustre: lustre-MDT0000: new disk, initializing [ 3717.374586] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3717.388535] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3721.964822] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 3732.063977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3732.212435] Lustre: 119665:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3732.288647] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3732.292798] Lustre: Skipped 1 previous similar message [ 3732.388993] Lustre: lustre-MDT0001: new disk, initializing [ 3732.491995] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3732.532054] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3732.545321] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3736.825672] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 3746.430435] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3751.858437] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3758.729125] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3759.005459] Lustre: lustre-OST0000: new disk, initializing [ 3759.009479] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3759.017029] Lustre: 121632:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3759.120247] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3765.049371] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 3767.347752] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3767.358109] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3767.449670] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3774.386420] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3778.755750] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3778.993196] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3782.996626] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3783.226769] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3788.257577] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3788.261133] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3788.274982] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3788.274991] Lustre: Skipped 1 previous similar message [ 3791.271951] Lustre: server umount lustre-OST0000 complete [ 3798.501119] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3798.524382] Lustre: Skipped 1 previous similar message [ 3798.528476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3798.534629] Lustre: Skipped 2 previous similar messages [ 3800.661138] Lustre: server umount lustre-MDT0000 complete [ 3803.629950] LustreError: 119678:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3803.655764] LustreError: 119678:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3805.213774] LustreError: 119658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782221307 with bad export cookie 11683936366230544808 [ 3805.233934] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3805.237201] LustreError: 119658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3805.551520] Lustre: server umount lustre-MDT0001 complete [ 3814.264892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3814.926481] LustreError: 123850:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3815.010335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3819.359199] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 3820.002866] LustreError: 123851:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3824.096604] LustreError: 123850:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3826.207216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3831.115888] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 3838.030935] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3843.348339] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3849.065042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3854.089859] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 3860.236509] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3866.798852] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3872.226065] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3872.239654] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3872.250492] Lustre: Skipped 1 previous similar message [ 3872.261672] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3875.302191] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3876.500804] Lustre: server umount lustre-OST0000 complete [ 3880.419646] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3880.438487] Lustre: Skipped 1 previous similar message [ 3880.448045] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3880.455922] Lustre: Skipped 2 previous similar messages [ 3894.239163] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3894.549093] Lustre: server umount lustre-MDT0000 complete [ 3895.778622] LustreError: 123845:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3895.801112] LustreError: 123845:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3898.254437] LustreError: 123831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782221400 with bad export cookie 11683936366230545690 [ 3898.260844] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3898.278572] LustreError: 123831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3898.689288] Lustre: server umount lustre-MDT0001 complete [ 3909.956706] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 3912.630978] Key type lgssc unregistered [ 3912.884691] LNet: 127478:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3912.894340] LNetError: 127478:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3912.915428] LNet: Removed LNI 192.168.204.116@tcp [ 3913.755393] Key type .llcrypt unregistered [ 3913.757221] Key type ._llcrypt unregistered [ 3931.158465] Key type ._llcrypt registered [ 3931.162572] Key type .llcrypt registered [ 3931.286583] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_hostid [ 3944.932746] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 3945.639588] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3945.652995] alg: No test for adler32 (adler32-zlib) [ 3946.709850] Lustre: Lustre: Build Version: 2.17.54_83_g7183f09 [ 3947.006874] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 3948.696826] Key type lgssc registered [ 3949.575846] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3985.754239] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3985.788387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3987.171172] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3987.210225] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3987.383928] Lustre: lustre-MDT0000: new disk, initializing [ 3987.548387] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3987.566325] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3993.201122] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 4003.698806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4003.802657] Lustre: 130634:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4003.861632] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4003.875450] Lustre: Skipped 1 previous similar message [ 4003.981837] Lustre: lustre-MDT0001: new disk, initializing [ 4004.108174] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4004.158252] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4004.169669] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4009.345923] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 4019.674263] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4025.540177] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4032.231968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4032.583628] Lustre: lustre-OST0000: new disk, initializing [ 4032.588620] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4032.595860] Lustre: 132603:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4032.705760] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4039.376560] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 4040.784137] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4040.796460] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4040.907605] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4048.910564] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4053.521443] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4053.758902] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4058.625422] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4058.822502] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4061.665366] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4061.666365] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4061.676083] LustreError: Skipped 1 previous similar message [ 4061.681447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4061.689978] Lustre: Skipped 1 previous similar message [ 4066.943546] Lustre: server umount lustre-OST0000 complete [ 4070.884109] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4070.894094] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4070.910709] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4070.919785] Lustre: Skipped 1 previous similar message [ 4071.908243] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4071.937949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4071.945973] Lustre: Skipped 1 previous similar message [ 4076.726392] Lustre: server umount lustre-MDT0000 complete [ 4077.026983] LustreError: 130644:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4077.043957] LustreError: 130644:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 4080.031622] LustreError: 130625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782221581 with bad export cookie 9037830121762260131 [ 4080.035963] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4080.037710] LustreError: 130625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4080.252322] Lustre: server umount lustre-MDT0001 complete [ 4087.439545] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 09:33:07 (1782221587) [ 4093.447044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4094.026963] LustreError: 135016:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4094.088465] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4097.733902] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 4099.555302] LustreError: 135017:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4103.651626] LustreError: 135016:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4104.725388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4105.044984] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4109.293451] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 4116.195429] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4122.363992] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4125.679191] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4125.686532] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4125.700718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4129.763643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4129.772282] Lustre: Skipped 3 previous similar messages [ 4131.203955] Lustre: server umount lustre-MDT0000 complete [ 4134.893659] LustreError: 136064:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4134.911698] LustreError: 136064:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4134.929689] LustreError: 134998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782221636 with bad export cookie 9037830121762261013 [ 4134.936538] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4134.940616] LustreError: 134998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4135.388621] Lustre: server umount lustre-MDT0001 complete [ 4143.667549] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 09:34:03 (1782221643) [ 4149.847761] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4165.408069] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4170.527570] LustreError: 137430:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.116@tcp: failed processing log, type 4: rc = -110 [ 4196.319331] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4202.155288] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 4209.496948] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4211.432880] Lustre: server umount lustre-OST0000 complete [ 4219.500110] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 09:35:19 (1782221719) [ 4225.144942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4225.512536] LustreError: 138874:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4225.587839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4230.095114] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 4238.061778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4242.625816] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 4250.083369] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4255.514984] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4262.095042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4262.443384] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4262.446201] Lustre: Skipped 1 previous similar message [ 4268.531927] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 4275.902512] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4284.387234] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4284.397374] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4284.405843] Lustre: Skipped 1 previous similar message [ 4284.408851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4288.492283] Lustre: server umount lustre-MDT0000 complete [ 4291.041133] LustreError: 141317:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4291.057712] LustreError: 141317:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 4291.726770] LustreError: 138854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782221793 with bad export cookie 9037830121762261671 [ 4291.730308] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4291.733758] LustreError: 138854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4292.104660] Lustre: server umount lustre-MDT0001 complete [ 4306.214925] Lustre: server umount lustre-OST0000 complete [ 4313.814050] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 09:36:54 (1782221814) [ 4320.016229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4325.190399] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 4325.857345] LustreError: 142529:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4325.872210] LustreError: 142529:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4332.933039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4333.210833] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4333.223273] Lustre: Skipped 1 previous similar message [ 4337.119887] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 4344.475278] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4349.768243] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4356.258902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4362.824762] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 4365.944268] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4369.894693] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4374.383859] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4374.646311] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4378.456164] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4378.706152] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4381.664347] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4381.676439] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4381.682240] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4381.708054] Lustre: Skipped 3 previous similar messages [ 4381.710564] Lustre: Skipped 3 previous similar messages [ 4386.726359] Lustre: server umount lustre-OST0000 complete [ 4404.191192] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4404.383094] Lustre: server umount lustre-MDT0000 complete [ 4406.240427] LustreError: 142528:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4406.267038] LustreError: 142528:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4408.107348] LustreError: 142510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782221910 with bad export cookie 9037830121762262742 [ 4408.112771] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4408.117616] LustreError: 142510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4408.574887] Lustre: server umount lustre-MDT0001 complete [ 4416.533330] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 09:38:36 (1782221916) [ 4422.923098] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4438.562244] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4443.681238] LustreError: 146688:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.116@tcp: failed processing log, type 4: rc = -110 [ 4469.407247] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4469.412155] Lustre: Skipped 1 previous similar message [ 4475.104993] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 4482.449453] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4488.368977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4488.824064] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4492.719202] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 4501.091695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4506.173559] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 4513.691807] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4518.803812] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4523.755292] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4524.034472] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4529.368658] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4529.676764] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4532.192785] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4532.203323] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4532.221079] Lustre: Skipped 2 previous similar messages [ 4532.225633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4532.228247] Lustre: Skipped 7 previous similar messages [ 4537.938847] Lustre: server umount lustre-MDT0000 complete [ 4540.391861] LustreError: 147725:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4540.403678] LustreError: 147725:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4541.439710] LustreError: 146696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782222043 with bad export cookie 9037830121762263771 [ 4541.447204] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4541.451179] LustreError: 146696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4541.880645] Lustre: server umount lustre-MDT0001 complete [ 4555.808787] Lustre: server umount lustre-OST0000 complete [ 4562.969435] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 09:41:03 (1782222063) [ 4568.472077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4584.096959] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4589.215480] LustreError: 150881:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.116@tcp: failed processing log, type 4: rc = -110 [ 4621.254077] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 4628.131900] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4633.821096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4634.330971] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4638.110793] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 4646.005323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4650.772902] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 4657.668880] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4663.521446] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4670.494119] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4670.733735] Lustre: lustre-OST0001: new disk, initializing [ 4670.739219] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4670.748558] Lustre: 153753:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4676.898684] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 4679.260710] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4679.272223] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4679.337520] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4685.382605] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4689.036852] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4689.225922] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4692.979925] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4693.160980] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4695.008164] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4695.023908] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4695.026151] Lustre: Skipped 3 previous similar messages [ 4695.033284] Lustre: Skipped 3 previous similar messages [ 4709.343163] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4709.439642] Lustre: server umount lustre-OST0000 complete [ 4715.508976] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4715.523207] LustreError: Skipped 1 previous similar message [ 4718.849618] Lustre: server umount lustre-OST0001 complete [ 4729.995427] Lustre: server umount lustre-MDT0000 complete [ 4734.435268] LustreError: 150888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782222236 with bad export cookie 9037830121762264723 [ 4734.460019] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4734.735061] Lustre: server umount lustre-MDT0001 complete [ 4756.171277] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 09:44:16 (1782222256) [ 4757.811986] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4759.357942] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4760.931297] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 09:44:21 (1782222261) [ 4767.047439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4767.378279] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4767.409905] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4767.592156] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4767.605787] Lustre: Skipped 6 previous similar messages [ 4771.978978] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 4780.019397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4780.081569] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4780.110152] Lustre: 157316:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4784.193705] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 4791.303278] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4796.362804] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4801.490547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4801.671908] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4807.114338] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 4809.318981] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4813.350279] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4818.107217] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4818.348232] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4822.772648] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4823.033955] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4825.070447] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4825.079777] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4825.081802] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4825.097728] Lustre: Skipped 5 previous similar messages [ 4825.119699] Lustre: Skipped 9 previous similar messages [ 4838.879175] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4839.102933] Lustre: server umount lustre-OST0000 complete [ 4860.629368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4866.036573] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4866.058596] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4866.060297] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4866.109373] Lustre: Skipped 1 previous similar message [ 4866.414686] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 4875.858776] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4882.999472] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4883.154807] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4888.249097] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4888.462348] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4894.039884] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9bf048d0b000.ost_server_uuid 50 [ 4895.882887] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9bf048d0b000.ost_server_uuid in FULL state after 0 sec [ 4896.093556] LustreError: 157322:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.204.16@tcp) returned error from blocking AST (req@00000000ccfd04e1 x1868794637318656 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 000000002719e7cc/0x7d6cd29c7bf71fed lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.16@tcp remote: 0x397cc7edd045ed6c expref: 5 pid: 159184 timeout: 4996 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4896.120158] LustreError: lustre-MDT0000: A client on nid 192.168.204.16@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4896.138481] LustreError: 157311:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.204.16@tcp ns: mdt-lustre-MDT0000_UUID lock: 000000002719e7cc/0x7d6cd29c7bf71fed lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.16@tcp remote: 0x397cc7edd045ed6c expref: 6 pid: 159184 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4901.857226] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4905.780703] Lustre: server umount lustre-OST0000 complete [ 4923.359180] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4923.565238] Lustre: server umount lustre-MDT0000 complete [ 4926.437366] LustreError: 157322:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4926.447056] LustreError: 157322:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 4926.873495] LustreError: 159175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782222428 with bad export cookie 9037830121762265906 [ 4926.890573] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4926.895136] LustreError: 159175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 4941.279464] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4941.513792] Lustre: server umount lustre-MDT0001 complete [ 4953.056757] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 4955.996574] Key type lgssc unregistered [ 4956.240842] LNet: 163429:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4956.252262] LNetError: 163429:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4957.285577] LNet: Removed LNI 192.168.204.116@tcp [ 4957.843667] Key type .llcrypt unregistered [ 4957.848098] Key type ._llcrypt unregistered [ 4979.478419] Key type ._llcrypt registered [ 4979.479817] Key type .llcrypt registered [ 4979.568924] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 4980.476431] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4980.529433] alg: No test for adler32 (adler32-zlib) [ 4981.547564] Lustre: Lustre: Build Version: 2.17.54_83_g7183f09 [ 4981.786994] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 4983.479187] Key type lgssc registered [ 4984.579955] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4992.446447] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4992.483342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4994.124748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4998.297957] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 5003.234057] LustreError: 164843:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5003.257346] LustreError: 164843:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5007.059022] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5007.460576] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5011.105729] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 5018.318284] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5025.592589] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5032.905272] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5033.422671] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5038.586628] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5039.236721] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 5046.094317] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5051.548993] Lustre: Failing over lustre-MDT0000 [ 5052.988645] LustreError: 164839:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5053.415065] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5053.427677] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5053.866883] Lustre: server umount lustre-MDT0000 complete [ 5059.041459] LustreError: 164844:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5059.056393] LustreError: 164844:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5064.163490] LustreError: 164843:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5064.177025] LustreError: 164843:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5066.019522] LustreError: 164823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.16@tcp arrived at 1782222567 with bad export cookie 598548678017651436 [ 5066.034312] LustreError: 164823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5069.220281] Lustre: 164154:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782222555/real 1782222555] req@0000000068d218d5 x1868795722359040/t0(0) o400->MGC192.168.204.116@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1782222571 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5069.261199] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5069.281342] LustreError: 165572:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5069.298453] LustreError: 165572:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5073.714332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5079.523342] LustreError: 164838:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5079.539939] LustreError: 164838:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5079.724760] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5079.768352] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5084.741078] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 5085.176297] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5085.187468] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5087.284553] Lustre: lustre-MDT0000: Denying connection for new client 8ef25739-c07d-41a6-9374-822530829563 (at 192.168.204.16@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 5095.397676] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5095.405776] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5095.417120] Lustre: Skipped 3 previous similar messages [ 5100.523179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5100.534628] Lustre: Skipped 2 previous similar messages [ 5101.492628] Lustre: server umount lustre-OST0000 complete [ 5104.563865] LustreError: 168577:0:(ldlm_lib.c:2990:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5104.570819] Lustre: 167926:0:(ldlm_lib.c:2390:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5104.580297] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5104.594165] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5104.602099] LustreError: 167926:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000daa6f8db x1868795722377472/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5104.613932] LustreError: 167926:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5104.620481] LustreError: 167926:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5104.628208] Lustre: lustre-MDT0000: Recovery over after 0:19, of 2 clients 0 recovered and 2 were evicted. [ 5104.918548] Lustre: server umount lustre-MDT0000 complete [ 5105.633106] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5105.644851] LustreError: 164840:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5108.378415] LustreError: 164824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782222610 with bad export cookie 598548678017651534 [ 5108.382166] LustreError: 164152:0:(import.c:709:ptlrpc_connect_import_locked()) can't connect to a closed import [ 5108.386942] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5108.389063] LustreError: 164824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5110.759987] Lustre: server umount lustre-MDT0001 complete [ 5123.081280] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 5126.395470] Key type lgssc unregistered [ 5126.676270] LNet: 169329:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5126.682269] LNetError: 169329:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5126.708545] LNet: Removed LNI 192.168.204.116@tcp [ 5127.663904] Key type .llcrypt unregistered [ 5127.671177] Key type ._llcrypt unregistered [ 5150.183205] Key type ._llcrypt registered [ 5150.186024] Key type .llcrypt registered [ 5150.250950] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 5151.256281] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5151.350597] alg: No test for adler32 (adler32-zlib) [ 5152.494199] Lustre: Lustre: Build Version: 2.17.54_83_g7183f09 [ 5152.735259] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 5154.415248] Key type lgssc registered [ 5156.236853] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5164.141327] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5164.157535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5165.614877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5170.402342] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 5175.781229] LustreError: 170742:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5175.804234] LustreError: 170742:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5179.955321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5180.326847] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5184.643187] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 5193.502549] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5201.445245] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5210.569190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5211.030174] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5217.240804] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 5224.943454] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5235.638534] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5245.929093] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5245.929525] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5245.941507] LustreError: Skipped 1 previous similar message [ 5245.942751] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5245.964184] Lustre: Skipped 1 previous similar message [ 5249.407483] Lustre: server umount lustre-OST0000 complete [ 5256.171353] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5256.178916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5256.191858] Lustre: Skipped 1 previous similar message [ 5256.209556] Lustre: Skipped 2 previous similar messages [ 5259.920644] Lustre: server umount lustre-MDT0000 complete [ 5261.285417] LustreError: 170741:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5261.313290] LustreError: 170741:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5263.714874] LustreError: 173544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782222765 with bad export cookie 2981796895950554048 [ 5263.724356] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5263.742460] LustreError: 173544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5264.139643] Lustre: server umount lustre-MDT0001 complete [ 5276.598404] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 5279.058632] Key type lgssc unregistered [ 5279.353963] LNet: 174295:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5279.362850] LNetError: 174295:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5279.398802] LNet: Removed LNI 192.168.204.116@tcp [ 5280.616223] Key type .llcrypt unregistered [ 5280.620667] Key type ._llcrypt unregistered [ 5312.500497] Key type ._llcrypt registered [ 5312.505851] Key type .llcrypt registered [ 5312.624748] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 5313.614492] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5313.720396] alg: No test for adler32 (adler32-zlib) [ 5314.763267] Lustre: Lustre: Build Version: 2.17.54_83_g7183f09 [ 5315.138355] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 5316.825203] Key type lgssc registered [ 5318.050617] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5325.806932] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5325.847805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5327.284781] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5331.599799] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 5337.572728] LustreError: 175998:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5337.600746] LustreError: 175998:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5339.568608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5339.971048] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5345.222676] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 5352.636519] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5357.717908] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5363.994045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5364.340396] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5370.611266] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 5372.541203] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5377.596540] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5386.212769] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5386.220286] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5386.234772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5386.703901] Lustre: server umount lustre-OST0000 complete [ 5391.331295] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5391.342584] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5391.367563] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5393.377091] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5393.377435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5393.413240] Lustre: Skipped 1 previous similar message [ 5398.498290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5398.507579] Lustre: Skipped 1 previous similar message [ 5399.614399] Lustre: server umount lustre-MDT0000 complete [ 5402.987847] LustreError: 175979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782222904 with bad export cookie 7687289695338111130 [ 5402.998888] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5403.001966] LustreError: 175979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5403.318665] Lustre: server umount lustre-MDT0001 complete [ 5414.500483] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 5417.547267] Key type lgssc unregistered [ 5417.875696] LNet: 179531:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5417.896471] LNetError: 179531:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5417.917727] LNet: Removed LNI 192.168.204.116@tcp [ 5418.905924] Key type .llcrypt unregistered [ 5418.908180] Key type ._llcrypt unregistered [ 5444.460647] Key type ._llcrypt registered [ 5444.462465] Key type .llcrypt registered [ 5444.580647] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 5445.604791] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5445.630612] alg: No test for adler32 (adler32-zlib) [ 5446.679762] Lustre: Lustre: Build Version: 2.17.54_83_g7183f09 [ 5447.059224] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 5448.776476] Key type lgssc registered [ 5449.833252] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5462.942469] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5462.963408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5464.527211] Lustre: *** cfs_fail_loc=135, val=0*** [ 5464.711129] LustreError: 181058:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5464.731603] LustreError: 181058:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.116@tcp: cfg command failed: rc = -2 [ 5464.750868] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5464.750868] [ 5464.773522] LustreError: MGC192.168.204.116@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5464.788333] LustreError: 181019:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5464.794604] LustreError: 181019:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5464.824227] LustreError: 181019:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 5464.829472] LustreError: 181019:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5464.929316] Lustre: server umount lustre-MDT0000 complete [ 5464.931135] LustreError: 181019:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5472.533643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5473.224883] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5477.754226] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 5483.493082] LustreError: 181431:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5483.502810] LustreError: 181431:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5486.620446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5487.051972] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5492.018541] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 5500.009825] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5505.087420] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5507.552777] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5507.569797] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5507.610198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5508.577307] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5508.579237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5508.615537] Lustre: Skipped 1 previous similar message [ 5511.713691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5511.726319] Lustre: Skipped 1 previous similar message [ 5516.768365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5516.779180] Lustre: Skipped 1 previous similar message [ 5521.376606] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5521.577590] Lustre: server umount lustre-MDT0000 complete [ 5521.896576] LustreError: 181425:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5521.922143] LustreError: 181425:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5524.964623] LustreError: 181411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782223026 with bad export cookie 13052152795052928636 [ 5524.969192] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5524.983817] LustreError: 181411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5525.369334] Lustre: server umount lustre-MDT0001 complete [ 5535.084744] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 5537.773370] Key type lgssc unregistered [ 5538.000986] LNet: 183751:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5538.012624] LNetError: 183751:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5538.023162] LNet: Removed LNI 192.168.204.116@tcp [ 5538.775231] Key type .llcrypt unregistered [ 5538.778283] Key type ._llcrypt unregistered [ 5556.648428] Key type ._llcrypt registered [ 5556.650434] Key type .llcrypt registered [ 5556.724587] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 5556.935339] Key type .llcrypt unregistered [ 5556.941545] Key type ._llcrypt unregistered [ 5571.923691] Key type ._llcrypt registered [ 5571.926646] Key type .llcrypt registered [ 5572.009240] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 5572.822194] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5572.892658] alg: No test for adler32 (adler32-zlib) [ 5573.906639] Lustre: Lustre: Build Version: 2.17.54_83_g7183f09 [ 5574.100199] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 5575.775857] Key type lgssc registered [ 5576.776642] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5585.681262] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5585.694935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5601.183343] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5606.305720] LustreError: 185883:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.116@tcp: failed processing log, type 4: rc = -110 [ 5632.991712] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5638.901959] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 5645.087781] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5650.237492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5650.694385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5650.760941] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5654.211406] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 5656.037536] LustreError: 185909:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5656.057252] LustreError: 185909:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5661.161487] LustreError: 185909:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5661.814331] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5662.188321] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5666.312873] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 5674.578497] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5680.302279] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5683.095368] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5696.992974] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5697.008796] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5697.029495] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5698.022492] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5698.026771] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5698.032980] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5698.055682] Lustre: Skipped 1 previous similar message [ 5699.295754] Lustre: server umount lustre-OST0000 complete [ 5703.138573] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5703.154716] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5703.156233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5703.165639] Lustre: Skipped 1 previous similar message [ 5708.259243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5708.267710] Lustre: Skipped 1 previous similar message [ 5709.456179] Lustre: server umount lustre-MDT0000 complete [ 5712.921409] LustreError: 185889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782223214 with bad export cookie 4259399441818961082 [ 5712.934255] LustreError: 185889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5712.939946] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5713.256204] Lustre: server umount lustre-MDT0001 complete [ 5723.109859] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 5725.564625] Key type lgssc unregistered [ 5725.850504] LNet: 189847:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5725.857263] LNetError: 189847:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5725.872733] LNet: Removed LNI 192.168.204.116@tcp [ 5726.566097] Key type .llcrypt unregistered [ 5726.570676] Key type ._llcrypt unregistered [ 5747.590828] Key type ._llcrypt registered [ 5747.593616] Key type .llcrypt registered [ 5747.681598] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 5748.437674] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5748.459699] alg: No test for adler32 (adler32-zlib) [ 5749.555278] Lustre: Lustre: Build Version: 2.17.54_83_g7183f09 [ 5749.808813] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 5751.527163] Key type lgssc registered [ 5752.637023] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5761.368658] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5761.388479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5762.890268] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5767.588308] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 5772.259417] LustreError: 191271:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5772.277397] LustreError: 191271:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5776.527126] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5776.875393] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5781.708570] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 5788.604450] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5793.156293] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5798.598106] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5798.870378] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5803.895955] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 5807.103406] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 5810.131943] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5815.379182] Lustre: Failing over lustre-MDT0000 [ 5815.645130] Lustre: server umount lustre-MDT0000 complete [ 5817.315437] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5817.327653] LustreError: 191267:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5817.332584] Lustre: Skipped 2 previous similar messages [ 5817.364747] LustreError: 191267:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5822.435071] LustreError: 191271:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5822.450558] LustreError: 191271:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5827.553837] LustreError: 191266:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5827.578336] LustreError: 191266:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5832.683315] LustreError: 191265:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5832.692960] LustreError: 191265:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5833.706124] Lustre: 190569:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782223319/real 1782223319] req@000000006e5aef79 x1868796527664384/t0(0) o400->MGC192.168.204.116@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1782223335 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5833.726923] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5834.544382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5843.479879] LustreError: 191266:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5843.513625] LustreError: 191266:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 5844.272972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5844.342556] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5848.605710] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5848.876693] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 5849.594833] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5849.622118] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 5849.655494] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 5853.202438] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 5860.861548] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 5876.717721] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5876.728776] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5876.753283] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5878.194553] Lustre: server umount lustre-OST0000 complete [ 5885.409903] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5885.410891] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5885.440659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5890.529297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5890.543131] Lustre: Skipped 2 previous similar messages [ 5894.111615] Lustre: 190569:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782223380/real 1782223380] req@00000000674648b5 x1868796527699328/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1782223396 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5894.148102] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5894.166973] Lustre: Skipped 1 previous similar message [ 5894.179834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5894.186812] Lustre: Skipped 1 previous similar message [ 5896.159274] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5896.358078] Lustre: server umount lustre-MDT0000 complete [ 5896.671109] Lustre: 190570:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782223382/real 1782223382] req@00000000b82cda41 x1868796527699712/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1782223398 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5899.237066] LustreError: 191271:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5899.274240] LustreError: 191271:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5900.108368] LustreError: 194849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782223402 with bad export cookie 16827695459058246998 [ 5900.117659] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5900.129182] LustreError: 194849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5900.568713] Lustre: server umount lustre-MDT0001 complete [ 5912.263916] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 5914.861591] Key type lgssc unregistered [ 5915.208025] LNet: 196187:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5915.217756] LNetError: 196187:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5915.245315] LNet: Removed LNI 192.168.204.116@tcp [ 5916.234204] Key type .llcrypt unregistered [ 5916.239068] Key type ._llcrypt unregistered [ 5939.058413] Key type ._llcrypt registered [ 5939.063538] Key type .llcrypt registered [ 5939.315344] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 5940.525514] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5940.643084] alg: No test for adler32 (adler32-zlib) [ 5942.059300] Lustre: Lustre: Build Version: 2.17.54_83_g7183f09 [ 5942.329285] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 5944.015177] Key type lgssc registered [ 5945.588779] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5955.166132] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5955.200674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5956.682112] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5961.729770] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 5966.816823] LustreError: 197598:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5966.840432] LustreError: 197598:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5970.002024] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5970.351966] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5974.419228] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 5981.262683] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5986.580163] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5993.010968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5993.449161] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5999.182779] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 6001.714224] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6006.518885] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6009.445503] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6018.044974] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6032.622468] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6040.562365] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6040.582945] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6040.612321] Lustre: Skipped 1 previous similar message [ 6040.643714] Lustre: Skipped 1 previous similar message [ 6044.446367] Lustre: server umount lustre-OST0000 complete [ 6050.799801] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6050.805438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6050.834034] Lustre: Skipped 1 previous similar message [ 6050.853541] Lustre: Skipped 1 previous similar message [ 6055.918577] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6055.929945] Lustre: Skipped 1 previous similar message [ 6057.140748] Lustre: server umount lustre-MDT0000 complete [ 6061.026436] LustreError: 198327:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6061.047561] LustreError: 198327:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6061.321525] LustreError: 197579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782223563 with bad export cookie 713633440914963405 [ 6061.328281] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6061.331180] LustreError: 197579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6061.689927] Lustre: server umount lustre-MDT0001 complete [ 6073.189290] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 6076.304348] Key type lgssc unregistered [ 6076.613774] LNet: 201447:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6076.625857] LNetError: 201447:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6076.648253] LNet: Removed LNI 192.168.204.116@tcp [ 6077.612914] Key type .llcrypt unregistered [ 6077.615216] Key type ._llcrypt unregistered [ 6097.953415] Key type ._llcrypt registered [ 6097.957407] Key type .llcrypt registered [ 6098.073224] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 6099.106219] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6099.133602] alg: No test for adler32 (adler32-zlib) [ 6100.283751] Lustre: Lustre: Build Version: 2.17.54_83_g7183f09 [ 6100.543075] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 6102.231667] Key type lgssc registered [ 6103.505450] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6111.627131] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6111.643971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6113.235721] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6117.852796] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 6123.492502] LustreError: 202714:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6123.537343] LustreError: 202714:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6127.287504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6127.701975] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6132.322526] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 6139.576223] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6145.678344] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6153.419271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6153.789292] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6158.897626] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6160.958281] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 6169.763685] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6176.385389] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6190.418697] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6201.120946] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6206.782855] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6218.732323] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6218.733299] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6218.749955] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6218.759457] Lustre: Skipped 1 previous similar message [ 6223.307714] Lustre: server umount lustre-OST0000 complete [ 6228.962557] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6228.964192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6228.986796] Lustre: Skipped 2 previous similar messages [ 6229.003066] Lustre: Skipped 1 previous similar message [ 6233.063051] Lustre: server umount lustre-MDT0000 complete [ 6234.084917] LustreError: 203442:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6234.104527] LustreError: 203442:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6237.026219] LustreError: 205391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782223738 with bad export cookie 15877659851834213943 [ 6237.031413] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6237.035849] LustreError: 205391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6237.452224] Lustre: server umount lustre-MDT0001 complete [ 6248.508805] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 6251.241249] Key type lgssc unregistered [ 6251.539446] LNet: 207734:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6251.551278] LNetError: 207734:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6251.575288] LNet: Removed LNI 192.168.204.116@tcp [ 6252.383746] Key type .llcrypt unregistered [ 6252.387097] Key type ._llcrypt unregistered [ 6276.332224] Key type ._llcrypt registered [ 6276.333840] Key type .llcrypt registered [ 6276.437622] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 6277.215380] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6277.280122] alg: No test for adler32 (adler32-zlib) [ 6278.339926] Lustre: Lustre: Build Version: 2.17.54_83_g7183f09 [ 6278.584253] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 6280.359230] Key type lgssc registered [ 6281.646369] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6289.122365] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6289.158570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6290.666602] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6295.062169] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 6300.129712] LustreError: 209149:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6300.146502] LustreError: 209149:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6303.989904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6304.377234] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6309.454727] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 6317.058444] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6322.511347] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6329.188770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6329.574501] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6335.743980] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6335.768144] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 6343.550586] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6360.550352] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6360.559496] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6360.565759] Lustre: Skipped 1 previous similar message [ 6360.586020] Lustre: Skipped 1 previous similar message [ 6365.667797] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6365.678721] Lustre: Skipped 1 previous similar message [ 6365.870404] Lustre: server umount lustre-OST0000 complete [ 6370.784267] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6370.787845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6370.807706] Lustre: Skipped 1 previous similar message [ 6370.838628] Lustre: Skipped 1 previous similar message [ 6375.906688] LustreError: 209145:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6375.926608] LustreError: 209145:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6376.030187] Lustre: server umount lustre-MDT0000 complete [ 6380.285552] LustreError: 209129:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782223882 with bad export cookie 9311585963620154909 [ 6380.292294] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6380.311548] LustreError: 209129:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6380.843844] Lustre: server umount lustre-MDT0001 complete [ 6393.711036] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 6397.172588] Key type lgssc unregistered [ 6397.570774] LNet: 212833:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6397.576480] LNetError: 212833:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6397.610818] LNet: Removed LNI 192.168.204.116@tcp [ 6398.738072] Key type .llcrypt unregistered [ 6398.742580] Key type ._llcrypt unregistered [ 6421.292561] Key type ._llcrypt registered [ 6421.294818] Key type .llcrypt registered [ 6421.412043] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 6422.725677] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6422.902963] alg: No test for adler32 (adler32-zlib) [ 6424.033362] Lustre: Lustre: Build Version: 2.17.54_83_g7183f09 [ 6424.378727] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 6426.111178] Key type lgssc registered [ 6427.531961] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6436.594105] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6436.649806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6438.140882] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6443.808917] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 6448.613150] LustreError: 214103:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6448.630290] LustreError: 214103:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6453.459982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6453.857948] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6459.155558] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 6466.677271] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6472.466832] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6480.002468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6480.449460] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6483.531239] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6487.168360] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 6494.858295] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6514.145256] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6514.151737] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6514.152800] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6514.168372] Lustre: Skipped 1 previous similar message [ 6517.590478] Lustre: server umount lustre-OST0000 complete [ 6524.393570] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6524.404107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6524.415039] Lustre: Skipped 1 previous similar message [ 6524.425979] Lustre: Skipped 2 previous similar messages [ 6528.228534] Lustre: server umount lustre-MDT0000 complete [ 6529.505622] LustreError: 214098:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6529.515761] LustreError: 214098:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6532.095622] LustreError: 214083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782224034 with bad export cookie 6063066831308119092 [ 6532.110477] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6532.636817] Lustre: server umount lustre-MDT0001 complete [ 6544.862293] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 6548.446909] Key type lgssc unregistered [ 6548.782137] LNet: 217786:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6548.798183] LNetError: 217786:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6549.862442] LNet: Removed LNI 192.168.204.116@tcp [ 6550.906129] Key type .llcrypt unregistered [ 6550.908196] Key type ._llcrypt unregistered [ 6570.712304] Key type ._llcrypt registered [ 6570.713690] Key type .llcrypt registered [ 6570.820283] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 6572.093131] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6572.166492] alg: No test for adler32 (adler32-zlib) [ 6573.309735] Lustre: Lustre: Build Version: 2.17.54_83_g7183f09 [ 6573.711040] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 6575.423172] Key type lgssc registered [ 6576.776950] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6585.384297] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6585.415378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6586.875229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6592.366647] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 6597.106314] LustreError: 219051:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6597.128918] LustreError: 219051:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6601.217803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6601.549841] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6606.659058] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 6614.828316] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6620.940835] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6627.446337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6627.914834] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6633.081417] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6634.252867] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 6641.869800] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6650.090103] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6650.162624] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6650.472446] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6658.547145] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 6667.504944] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6669.853838] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6681.713853] Lustre: Permanently deactivating lustre-OST0001 [ 6681.742926] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6687.323541] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6693.662348] Lustre: Permanently reactivating lustre-OST0001 [ 6693.676757] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6693.682332] Lustre: Skipped 2 previous similar messages [ 6696.050760] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6696.061528] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6696.074260] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6696.101963] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6696.149793] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6708.211644] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6708.220956] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6708.257811] Lustre: Skipped 2 previous similar messages [ 6708.275885] Lustre: Skipped 1 previous similar message [ 6713.316782] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6713.324076] Lustre: Skipped 1 previous similar message [ 6714.130233] Lustre: server umount lustre-OST0001 complete [ 6718.436426] LustreError: 222173:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6718.441541] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6718.442548] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6718.443474] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6718.463028] LustreError: 222173:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6723.561897] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6723.572135] Lustre: Skipped 2 previous similar messages [ 6723.576636] LustreError: 220875:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6723.608795] LustreError: 220875:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6723.966562] Lustre: server umount lustre-OST0000 complete [ 6728.672139] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6728.695295] Lustre: Skipped 1 previous similar message [ 6728.702210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6728.718613] Lustre: Skipped 1 previous similar message [ 6734.511655] Lustre: server umount lustre-MDT0000 complete [ 6738.699459] LustreError: 219033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782224240 with bad export cookie 4973616623468038094 [ 6738.712941] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6738.725287] LustreError: 219033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6739.370727] Lustre: server umount lustre-MDT0001 complete [ 6751.293730] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 6754.466156] Key type lgssc unregistered [ 6754.836122] LNet: 224120:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6754.853636] LNetError: 224120:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6754.872676] LNet: Removed LNI 192.168.204.116@tcp [ 6755.830230] Key type .llcrypt unregistered [ 6755.832146] Key type ._llcrypt unregistered [ 6798.554503] Key type ._llcrypt registered [ 6798.557345] Key type .llcrypt registered [ 6798.677713] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 6799.902036] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6799.957958] alg: No test for adler32 (adler32-zlib) [ 6801.125995] Lustre: Lustre: Build Version: 2.17.54_83_g7183f09 [ 6801.406790] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 6803.119183] Key type lgssc registered [ 6804.459501] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6813.437833] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6813.451987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6814.695935] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6814.728533] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6814.948422] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6819.868561] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 6828.380217] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6828.606514] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6828.651319] Lustre: 226327:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6828.995350] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6833.701259] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 6841.728186] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6847.706551] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6855.006955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6855.244048] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6855.546813] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6861.876761] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 6865.580053] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 6869.369120] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6873.242595] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6879.800140] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6885.700223] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6893.970656] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6906.303338] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6906.311181] Lustre: Skipped 1 previous similar message [ 6930.015873] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6930.025306] Lustre: Skipped 2 previous similar messages [ 6963.618680] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6963.631635] Lustre: Skipped 3 previous similar messages [ 7034.341251] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7034.353877] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7034.375040] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7038.847952] Lustre: server umount lustre-OST0000 complete [ 7044.064319] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7044.073770] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7044.087877] Lustre: Skipped 1 previous similar message [ 7044.090660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7044.102055] Lustre: Skipped 1 previous similar message [ 7045.094890] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7045.115791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7045.126197] Lustre: Skipped 1 previous similar message [ 7050.211512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7050.221086] Lustre: Skipped 1 previous similar message [ 7054.312130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7054.322184] Lustre: Skipped 1 previous similar message [ 7057.375479] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7057.608270] Lustre: server umount lustre-MDT0000 complete [ 7061.310248] LustreError: 228187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782224563 with bad export cookie 17562244086742502990 [ 7061.321629] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7061.337199] LustreError: 228187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7061.660955] Lustre: server umount lustre-MDT0001 complete [ 7074.206410] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 7077.787761] Key type lgssc unregistered [ 7078.189439] LNet: 231122:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7078.199899] LNetError: 231122:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7078.230988] LNet: Removed LNI 192.168.204.116@tcp [ 7079.135915] Key type .llcrypt unregistered [ 7079.141846] Key type ._llcrypt unregistered [ 7103.698779] Key type ._llcrypt registered [ 7103.702147] Key type .llcrypt registered [ 7103.781230] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 7105.039959] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7105.111732] alg: No test for adler32 (adler32-zlib) [ 7106.241543] Lustre: Lustre: Build Version: 2.17.54_83_g7183f09 [ 7106.651819] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 7108.367260] Key type lgssc registered [ 7109.834326] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7119.879795] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7119.906377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7121.534792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7127.035221] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 7131.108763] LustreError: 232536:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7131.118620] LustreError: 232536:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7134.453556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7134.907182] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7139.258391] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 7145.881746] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7152.055549] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7158.984839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7159.397580] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7165.470870] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 7166.588359] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7173.499951] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7191.525335] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7191.531257] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7191.539606] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7192.547258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7192.547767] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7192.548591] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7192.560882] Lustre: Skipped 1 previous similar message [ 7197.718506] Lustre: server umount lustre-OST0000 complete [ 7201.760285] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7201.777148] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7201.793994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7207.803211] Lustre: server umount lustre-MDT0000 complete [ 7207.910285] LustreError: 232532:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7207.934265] LustreError: 232532:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7212.001302] LustreError: 232518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782224713 with bad export cookie 6234121069348659115 [ 7212.008314] LustreError: MGC192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7212.014658] LustreError: 232518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7212.371778] Lustre: server umount lustre-MDT0001 complete [ 7225.292300] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 7228.358710] Key type lgssc unregistered [ 7228.726285] LNet: 236226:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7228.731991] LNetError: 236226:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7229.802752] LNet: Removed LNI 192.168.204.116@tcp [ 7231.092260] Key type .llcrypt unregistered [ 7231.100101] Key type ._llcrypt unregistered [ 7253.415327] Key type ._llcrypt registered [ 7253.420576] Key type .llcrypt registered [ 7253.579995] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 7254.964291] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7255.002044] alg: No test for adler32 (adler32-zlib) [ 7256.084150] Lustre: Lustre: Build Version: 2.17.54_83_g7183f09 [ 7256.371578] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 7258.120070] Key type lgssc registered [ 7259.273442] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7529.477983] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 7532.807404] Key type lgssc unregistered [ 7533.182939] LNet: 238257:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7533.198552] LNetError: 238257:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7533.219137] LNet: Removed LNI 192.168.204.116@tcp [ 7534.151415] Key type .llcrypt unregistered [ 7534.153319] Key type ._llcrypt unregistered [ 7562.072614] Key type ._llcrypt registered [ 7562.082449] Key type .llcrypt registered [ 7562.183457] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_hostid [ 7578.203796] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 7579.156275] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7579.375176] alg: No test for adler32 (adler32-zlib) [ 7580.405680] Lustre: Lustre: Build Version: 2.17.54_83_g7183f09 [ 7580.584679] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 7582.240967] Key type lgssc registered [ 7583.491689] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7615.145330] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7328 sec ========== 10:31:55 (1782225115) [ 7617.049230] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 10:31:56 (1782225116) === [ 7621.104285] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 10:32:00 (1782225120) === [ 7661.797606] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 7666.106270] Key type lgssc unregistered [ 7666.534324] LNet: 243490:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7666.571434] LNetError: 243490:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7666.595665] LNet: Removed LNI 192.168.204.116@tcp [ 7667.591472] Key type .llcrypt unregistered [ 7667.600080] Key type ._llcrypt unregistered