[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 469593447 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001015] APIC: Switch to symmetric I/O mode setup [ 0.003373] x2apic enabled [ 0.004015] Switched APIC routing to physical x2apic. [ 0.005021] kvm-guest: setup PV IPIs [ 0.007811] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008029] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009036] pid_max: default: 32768 minimum: 301 [ 0.010161] LSM: Security Framework initializing [ 0.011077] Yama: becoming mindful. [ 0.012049] SELinux: Initializing. [ 0.013095] *** VALIDATE selinux *** [ 0.021819] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026596] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028036] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029147] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030148] *** VALIDATE tmpfs *** [ 0.032195] *** VALIDATE proc *** [ 0.033282] *** VALIDATE cgroup *** [ 0.034016] *** VALIDATE cgroup2 *** [ 0.035318] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036192] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040045] Spectre V2 : User space: Vulnerable [ 0.041016] Speculative Store Bypass: Vulnerable [ 0.044636] debug: unmapping init [mem 0xffffffffb5659000-0xffffffffb5660fff] [ 0.046161] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047666] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048028] ... version: 2 [ 0.049020] ... bit width: 48 [ 0.050018] ... generic registers: 4 [ 0.051012] ... value mask: 0000ffffffffffff [ 0.052014] ... max period: 00007fffffffffff [ 0.053015] ... fixed-purpose events: 3 [ 0.054014] ... event mask: 000000070000000f [ 0.055304] rcu: Hierarchical SRCU implementation. [ 0.057303] smp: Bringing up secondary CPUs ... [ 0.058511] x86: Booting SMP configuration: [ 0.059024] .... node #0, CPUs: #1 #2 #3 [ 0.062442] smp: Brought up 1 node, 4 CPUs [ 0.063940] smpboot: Max logical packages: 1 [ 0.064020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.231030] node 0 deferred pages initialised in 164ms [ 0.235158] devtmpfs: initialized [ 0.236329] x86/mm: Memory block size: 128MB [ 0.239915] gcov: version magic: 0x41383552 [ 0.242316] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.246126] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.249364] pinctrl core: initialized pinctrl subsystem [ 0.251226] [ 0.251845] ************************************************************* [ 0.254019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.257033] ** ** [ 0.260016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.262013] ** ** [ 0.264016] ** This means that this kernel is built to expose internal ** [ 0.267019] ** IOMMU data structures, which may compromise security on ** [ 0.269013] ** your system. ** [ 0.271017] ** ** [ 0.274016] ** If you see this message and you are not debugging the ** [ 0.277017] ** kernel, report this immediately to your vendor! ** [ 0.279014] ** ** [ 0.282016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.284016] ************************************************************* [ 0.287742] NET: Registered protocol family 16 [ 0.289476] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.292074] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.295078] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.299039] cpuidle: using governor menu [ 0.300864] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.304659] PCI: Using configuration type 1 for base access [ 0.307138] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.318137] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.319036] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.320178] cryptd: max_cpu_qlen set to 1000 [ 0.324162] ACPI: Added _OSI(Module Device) [ 0.325018] ACPI: Added _OSI(Processor Device) [ 0.327015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.329019] ACPI: Added _OSI(Processor Aggregator Device) [ 0.333519] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.339555] ACPI: Interpreter enabled [ 0.341078] ACPI: PM: (supports S0 S3 S4 S5) [ 0.342016] ACPI: Using IOAPIC for interrupt routing [ 0.344155] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.348500] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.360026] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.361049] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.363023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.366104] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.371549] acpiphp: Slot [2] registered [ 0.373225] acpiphp: Slot [5] registered [ 0.375206] acpiphp: Slot [6] registered [ 0.376164] acpiphp: Slot [7] registered [ 0.378187] acpiphp: Slot [8] registered [ 0.379179] acpiphp: Slot [9] registered [ 0.381208] acpiphp: Slot [10] registered [ 0.382210] acpiphp: Slot [3] registered [ 0.384153] acpiphp: Slot [4] registered [ 0.385114] acpiphp: Slot [11] registered [ 0.386117] acpiphp: Slot [12] registered [ 0.388144] acpiphp: Slot [13] registered [ 0.390184] acpiphp: Slot [14] registered [ 0.391129] acpiphp: Slot [15] registered [ 0.392094] acpiphp: Slot [16] registered [ 0.394128] acpiphp: Slot [17] registered [ 0.395106] acpiphp: Slot [18] registered [ 0.396090] acpiphp: Slot [19] registered [ 0.397163] acpiphp: Slot [20] registered [ 0.399182] acpiphp: Slot [21] registered [ 0.400113] acpiphp: Slot [22] registered [ 0.402092] acpiphp: Slot [23] registered [ 0.403097] acpiphp: Slot [24] registered [ 0.404112] acpiphp: Slot [25] registered [ 0.405124] acpiphp: Slot [26] registered [ 0.407170] acpiphp: Slot [27] registered [ 0.409142] acpiphp: Slot [28] registered [ 0.410135] acpiphp: Slot [29] registered [ 0.412164] acpiphp: Slot [30] registered [ 0.414136] acpiphp: Slot [31] registered [ 0.415086] PCI host bridge to bus 0000:00 [ 0.417029] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.420033] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.422029] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.425034] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.428042] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.431034] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.433296] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.437151] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.440455] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.452015] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.456014] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.459057] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.461021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.463023] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.466562] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.469845] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.473065] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.476932] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.482023] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.495030] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.500023] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.507050] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.521023] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.531022] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.554026] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.564848] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.577023] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.586023] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.602026] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.617251] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.627023] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.637032] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.660023] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.673147] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.679024] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.685021] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.700029] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.712469] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.732023] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.749037] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.773020] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.784019] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.796021] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.804020] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.820024] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.833426] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.835389] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.837294] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.838382] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.841253] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.847121] iommu: Default domain type: Passthrough [ 0.849549] SCSI subsystem initialized [ 0.851203] ACPI: bus type USB registered [ 0.853151] usbcore: registered new interface driver usbfs [ 0.855111] usbcore: registered new interface driver hub [ 0.857126] usbcore: registered new device driver usb [ 0.859199] pps_core: LinuxPPS API ver. 1 registered [ 0.861013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.865076] PTP clock support registered [ 0.867108] EDAC MC: Ver: 3.0.0 [ 0.868544] PCI: Using ACPI for IRQ routing [ 0.871129] NetLabel: Initializing [ 0.873014] NetLabel: domain hash size = 128 [ 0.874013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.876104] NetLabel: unlabeled traffic allowed by default [ 0.879374] vgaarb: loaded [ 0.880437] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.882019] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.889000] clocksource: Switched to clocksource kvm-clock [ 0.999826] VFS: Disk quotas dquot_6.6.0 [ 1.001298] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.003354] *** VALIDATE ramfs *** [ 1.004379] *** VALIDATE hugetlbfs *** [ 1.005701] pnp: PnP ACPI init [ 1.008152] pnp: PnP ACPI: found 6 devices [ 1.025754] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.029183] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.031325] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.033329] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.036344] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.039200] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.042535] NET: Registered protocol family 2 [ 1.045671] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.051701] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.055730] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.080849] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.084732] TCP: Hash tables configured (established 65536 bind 65536) [ 1.088317] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.092227] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.095453] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.098726] NET: Registered protocol family 1 [ 1.102903] RPC: Registered named UNIX socket transport module. [ 1.105485] RPC: Registered udp transport module. [ 1.107436] RPC: Registered tcp transport module. [ 1.109293] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.111974] NET: Registered protocol family 44 [ 1.113344] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.115732] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.117801] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.120213] PCI: CLS 0 bytes, default 64 [ 1.121791] Unpacking initramfs... [ 2.497565] debug: unmapping init [mem 0xffff962f3cc54000-0xffff962f3ffbffff] [ 2.500848] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.502985] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.506170] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.032070] Initialise system trusted keyrings [ 3.034038] Key type blacklist registered [ 3.037621] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.046858] zbud: loaded [ 3.050377] *** VALIDATE nfs *** [ 3.052242] *** VALIDATE nfs4 *** [ 3.055919] pstore: using deflate compression [ 3.060695] Platform Keyring initialized [ 3.179868] NET: Registered protocol family 38 [ 3.181379] Key type asymmetric registered [ 3.182714] Asymmetric key parser 'x509' registered [ 3.183842] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.186041] io scheduler mq-deadline registered [ 3.187956] io scheduler kyber registered [ 3.189770] io scheduler bfq registered [ 3.191792] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.195144] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.197428] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.199726] ACPI: Power Button [PWRF] [ 3.206266] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.211843] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.222427] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.228960] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.244809] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.272354] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.303146] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.310574] Non-volatile memory driver v1.3 [ 3.312414] Linux agpgart interface v0.103 [ 3.351569] virtio_blk virtio1: [vda] 145896 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.356508] vda: detected capacity change from 0 to 74698752 [ 3.376992] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.380315] vdb: detected capacity change from 0 to 1073741824 [ 3.398311] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.401430] vdc: detected capacity change from 0 to 2621440000 [ 3.418686] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.421937] vdd: detected capacity change from 0 to 2621440000 [ 3.437563] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.441033] vde: detected capacity change from 0 to 4294967296 [ 3.458845] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.462300] vdf: detected capacity change from 0 to 4294967296 [ 3.472115] libphy: Fixed MDIO Bus: probed [ 3.478802] usbcore: registered new interface driver usbserial_generic [ 3.481846] usbserial: USB Serial support registered for generic [ 3.484477] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.489310] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.491505] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.494341] mousedev: PS/2 mouse device common for all mice [ 3.497606] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.499829] rtc_cmos 00:05: RTC can wake from S4 [ 3.503947] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.504373] rtc_cmos 00:05: registered as rtc0 [ 3.509031] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.509689] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.515617] intel_pstate: CPU model not supported [ 3.518649] hid: raw HID events driver (C) Jiri Kosina [ 3.521078] usbcore: registered new interface driver usbhid [ 3.523282] usbhid: USB HID core driver [ 3.524786] drop_monitor: Initializing network drop monitor service [ 3.526783] Initializing XFRM netlink socket [ 3.528467] NET: Registered protocol family 10 [ 3.531347] Segment Routing with IPv6 [ 3.532807] NET: Registered protocol family 17 [ 3.535548] mpls_gso: MPLS GSO support [ 3.540662] RAS: Correctable Errors collector initialized. [ 3.542986] AVX version of gcm_enc/dec engaged. [ 3.544776] AES CTR mode by8 optimization enabled [ 3.643842] sched_clock: Marking stable (3643790441, 0)->(4614447241, -970656800) [ 3.648148] registered taskstats version 1 [ 3.650173] Loading compiled-in X.509 certificates [ 3.652914] zswap: loaded using pool lzo/zbud [ 3.678764] Key type big_key registered [ 3.691483] Key type encrypted registered [ 3.692697] ima: No TPM chip found, activating TPM-bypass! [ 3.694291] ima: Allocated hash algorithm: sha1 [ 3.695388] ima: No architecture policies found [ 3.696565] evm: Initialising EVM extended attributes: [ 3.697909] evm: security.selinux [ 3.698709] evm: security.ima [ 3.699448] evm: security.capability [ 3.700373] evm: HMAC attrs: 0x1 [ 3.702376] rtc_cmos 00:05: setting system clock to 2026-09-03 13:07:36 UTC (1788440856) [ 3.706548] debug: unmapping init [mem 0xffffffffb6603000-0xffffffffb67fffff] [ 3.708525] debug: unmapping init [mem 0xffffffffb5382000-0xffffffffb5658fff] [ 3.715237] Write protecting the kernel read-only data: 28672k [ 3.718995] debug: unmapping init [mem 0xffffffffb3a03000-0xffffffffb3bfffff] [ 3.722247] debug: unmapping init [mem 0xffffffffb4314000-0xffffffffb43fffff] [ 3.770716] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.777921] systemd[1]: Detected virtualization kvm. [ 3.779095] systemd[1]: Detected architecture x86-64. [ 3.780541] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.806830] systemd[1]: No hostname configured. [ 3.808183] systemd[1]: Set hostname to . [ 3.809557] random: systemd: uninitialized urandom read (16 bytes read) [ 3.811228] systemd[1]: Initializing machine ID from random generator. [ 3.873839] random: ln: uninitialized urandom read (6 bytes read) [ 3.966514] random: systemd: uninitialized urandom read (16 bytes read) [ 3.969179] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.973900] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.979210] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. Starting Journal Service... [ OK ] Reached target Initrd Root Device. Starting Setup Virtual Console... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.584240] device-mapper: uevent: version 1.0.3 [ 4.585739] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.339356] virtio_net virtio0 ens2: renamed from eth0 [ 5.363099] random: fast init done [ 5.486851] scsi host0: ata_piix [ 5.512690] scsi host1: ata_piix [ 5.515078] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.517468] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.192529] dracut-initqueue[578]: RTNETLINK answers: File exists [ 10.204894] random: crng init done [ 10.206508] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.669192] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.774720] printk: systemd: 25 output lines suppressed due to ratelimiting [ 12.039272] SELinux: Disabled at runtime. [ 12.103267] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.112149] systemd[1]: Detected virtualization kvm. [ 12.114221] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.631276] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.634611] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.639248] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.643406] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.647225] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.656189] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.677402] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting POSIX Message Queue File System... Starting Apply Kernel Variables... [ OK ] Created slice User and Session Slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ 12.723628] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Initrd File Systems. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. Mounting Kernel Debug File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-getty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Huge Pages File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 13.065074] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.358810] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.359756] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.504220] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.519667] EDAC sbridge: Ver: 1.1.2 [ 15.114380] Key type dns_resolver registered [ 15.469610] NFS: Registering the id_resolver key type [ 15.471366] Key type id_resolver registered [ 15.473087] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg216-server login: [ 60.761699] hrtimer: interrupt took 5061676 ns [ 83.099639] libcfs: loading out-of-tree module taints kernel. [ 83.219514] Key type ._llcrypt registered [ 83.223719] Key type .llcrypt registered [ 83.362934] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 103.697199] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 105.554373] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 105.566708] alg: No test for adler32 (adler32-zlib) [ 107.175764] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 108.131332] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 109.888135] Key type lgssc registered [ 111.929056] Lustre: Echo OBD driver; http://www.lustre.org/ [ 128.608913] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 174.252596] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 189.986288] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 190.033206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 191.390672] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 191.416787] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 191.565730] Lustre: lustre-MDT0000: new disk, initializing [ 191.661337] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 191.676970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 196.833837] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 213.129270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 213.245928] Lustre: 6509:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 213.274838] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 213.278694] Lustre: Skipped 1 previous similar message [ 213.369840] Lustre: lustre-MDT0001: new disk, initializing [ 213.457603] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 213.490333] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 213.512914] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 218.927930] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 223.993450] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 233.742448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 234.035087] Lustre: lustre-OST0000: new disk, initializing [ 234.039182] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 234.047222] Lustre: 8445:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 234.127761] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 241.067560] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 241.181484] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 241.192247] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 241.320921] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 260.382889] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 260.645401] Lustre: lustre-OST0001: new disk, initializing [ 260.660332] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 260.685275] Lustre: 9520:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 260.769737] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 269.042675] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 270.376746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 270.384909] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 270.479077] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 284.741573] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 292.794853] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 301.481040] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing check_logdir /tmp/testlogs/ [ 307.764841] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing yml_node [ 311.875074] Lustre: DEBUG MARKER: Client: 2.17.56.51 [ 315.953695] Lustre: DEBUG MARKER: MDS: 2.17.56.51 [ 319.326358] Lustre: DEBUG MARKER: OSS: 2.17.56.51 [ 321.182439] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 3 09:12:52 EDT 2026 [ 340.420604] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 342.345548] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 361.957579] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 361.969732] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 361.981033] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 362.474940] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 362.489991] Lustre: Skipped 1 previous similar message [ 362.493109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 362.495794] Lustre: Skipped 2 previous similar messages [ 365.029749] Lustre: server umount lustre-MDT0000 complete [ 368.885865] LustreError: 6501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441221 with bad export cookie 3799855465513054413 [ 368.890926] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 368.894809] LustreError: 6501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 372.704556] LustreError: 6516:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 372.706087] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 372.710035] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 372.734806] LustreError: 6516:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 372.787600] Lustre: Skipped 2 previous similar messages [ 373.740425] LustreError: 6514:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 373.757692] LustreError: 6514:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 375.404790] Lustre: server umount lustre-MDT0001 complete [ 379.162790] Lustre: server umount lustre-OST0000 complete [ 382.985578] Lustre: server umount lustre-OST0001 complete [ 390.877817] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 401.428517] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 441.907553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 442.210328] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 442.260547] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 442.346943] Lustre: lustre-MDT0000: new disk, initializing [ 442.450240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 442.467120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 446.931620] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 457.496176] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 457.602957] Lustre: 15077:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 457.652601] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 457.655291] Lustre: Skipped 1 previous similar message [ 457.730353] Lustre: lustre-MDT0001: new disk, initializing [ 457.893199] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 458.005420] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 458.022498] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 462.143310] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 473.864735] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 480.635125] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 488.403537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 488.642595] Lustre: lustre-OST0000: new disk, initializing [ 488.646846] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 488.652745] Lustre: 17044:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 488.750572] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 489.908779] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 489.931344] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 490.084240] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 496.364903] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 508.532403] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 513.971874] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 514.231961] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 518.869697] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 519.130910] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 525.806566] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 525.824304] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 525.845678] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 525.847102] LustreError: Skipped 1 previous similar message [ 525.865315] Lustre: Skipped 4 previous similar messages [ 527.149705] Lustre: server umount lustre-OST0000 complete [ 534.504221] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 534.510196] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 534.527944] Lustre: Skipped 1 previous similar message [ 534.531836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 537.690702] Lustre: server umount lustre-MDT0000 complete [ 541.154519] LustreError: 15088:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 541.180682] LustreError: 15088:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 541.843052] LustreError: 15067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441394 with bad export cookie 3799855465513055939 [ 541.850272] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 541.857048] LustreError: 15067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 542.321950] Lustre: server umount lustre-MDT0001 complete [ 545.889383] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 09:16:37 (1788441397) [ 553.332268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 553.797328] LustreError: 19312:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 553.894907] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 558.425531] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 559.079760] LustreError: 19313:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 567.339778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 572.446626] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 581.040458] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 587.818780] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 594.933909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 602.593084] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 610.908520] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 623.089862] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 623.092963] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 623.110763] LustreError: Skipped 1 previous similar message [ 623.120592] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 623.143790] Lustre: Skipped 2 previous similar messages [ 623.183836] Lustre: Skipped 3 previous similar messages [ 628.611691] Lustre: server umount lustre-OST0000 complete [ 639.155778] Lustre: server umount lustre-MDT0000 complete [ 643.147338] LustreError: 19293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441495 with bad export cookie 3799855465513056828 [ 643.163747] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 643.169924] LustreError: 19293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 643.677262] Lustre: server umount lustre-MDT0001 complete [ 656.646668] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 659.480827] Key type lgssc unregistered [ 659.769812] LNet: 22849:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 659.775967] LNetError: 22849:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 660.845983] LNet: Removed LNI 192.168.202.116@tcp [ 661.842159] Key type .llcrypt unregistered [ 661.846315] Key type ._llcrypt unregistered [ 686.884804] Key type ._llcrypt registered [ 686.886945] Key type .llcrypt registered [ 686.969685] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 687.946621] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 688.061070] alg: No test for adler32 (adler32-zlib) [ 689.467919] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 689.992316] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 691.752512] Key type lgssc registered [ 693.375881] Lustre: Echo OBD driver; http://www.lustre.org/ [ 705.221331] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 705.245973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 707.096320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 712.864619] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 716.266819] LustreError: 24269:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 716.300502] LustreError: 24269:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 721.396387] LustreError: 24270:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 724.300320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 724.871852] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 730.581443] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 740.314565] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 746.428124] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 753.477715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 753.923767] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 760.266387] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 769.113595] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 776.449808] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 785.888514] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 785.894943] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 785.908193] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 786.922322] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 786.927219] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 786.942872] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 791.875963] Lustre: server umount lustre-OST0000 complete [ 796.642784] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 796.656969] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 796.674652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 796.686022] Lustre: Skipped 1 previous similar message [ 802.274207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 802.283678] Lustre: Skipped 3 previous similar messages [ 802.750332] Lustre: server umount lustre-MDT0000 complete [ 806.381693] LustreError: 24251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441659 with bad export cookie 4878959383160580227 [ 806.390059] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 806.392096] LustreError: 24251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 806.762799] Lustre: server umount lustre-MDT0001 complete [ 819.262219] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 822.433503] Key type lgssc unregistered [ 822.816918] LNet: 28248:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 822.822453] LNetError: 28248:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 823.850874] LNet: Removed LNI 192.168.202.116@tcp [ 824.877688] Key type .llcrypt unregistered [ 824.881146] Key type ._llcrypt unregistered [ 852.726873] Key type ._llcrypt registered [ 852.729637] Key type .llcrypt registered [ 852.838319] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 854.200044] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 854.354073] alg: No test for adler32 (adler32-zlib) [ 855.533566] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 855.863969] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 857.535203] Key type lgssc registered [ 859.220105] Lustre: Echo OBD driver; http://www.lustre.org/ [ 868.437529] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 868.465343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 870.148852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 875.432813] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 879.591691] LustreError: 29663:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 879.613855] LustreError: 29663:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 884.706353] LustreError: 29662:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 885.646483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 886.164087] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 891.265489] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 901.089748] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 907.920237] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 924.660160] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 925.132312] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 932.108566] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 939.443506] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 941.161739] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 949.728325] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 949.738427] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 949.745595] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 949.751937] Lustre: Skipped 1 previous similar message [ 949.758962] Lustre: Skipped 1 previous similar message [ 954.559215] Lustre: server umount lustre-OST0000 complete [ 959.972147] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 959.986274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 959.999725] Lustre: Skipped 1 previous similar message [ 964.879036] Lustre: server umount lustre-MDT0000 complete [ 965.096668] LustreError: 30650:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 965.127971] LustreError: 30650:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 968.826547] LustreError: 29642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441821 with bad export cookie 14592601537466646031 [ 968.832283] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 968.837592] LustreError: 29642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 969.170512] Lustre: server umount lustre-MDT0001 complete [ 981.357759] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 984.650397] Key type lgssc unregistered [ 984.941505] LNet: 33548:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 984.971563] LNetError: 33548:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 986.023260] LNet: Removed LNI 192.168.202.116@tcp [ 987.217189] Key type .llcrypt unregistered [ 987.222911] Key type ._llcrypt unregistered [ 1015.851281] Key type ._llcrypt registered [ 1015.863707] Key type .llcrypt registered [ 1016.091413] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 1017.860726] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1017.963142] alg: No test for adler32 (adler32-zlib) [ 1018.951839] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1019.160092] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 1020.863157] Key type lgssc registered [ 1022.603564] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1034.480590] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1034.530381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1036.594806] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1041.770724] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1047.021463] LustreError: 34964:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1047.045890] LustreError: 34964:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1050.585174] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1051.094798] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1055.964268] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1065.634893] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1072.842502] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1081.812084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1082.377443] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1090.752366] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1099.280428] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1106.968126] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1119.716986] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 1119.722396] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1119.732708] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1121.251899] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1121.265250] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1121.290771] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1125.497319] Lustre: server umount lustre-OST0000 complete [ 1132.513238] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1132.531541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1132.533494] Lustre: Skipped 2 previous similar messages [ 1136.367867] Lustre: server umount lustre-MDT0000 complete [ 1137.639061] LustreError: 34965:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1137.669743] LustreError: 34965:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1140.958088] LustreError: 34946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441993 with bad export cookie 7894489077440654863 [ 1140.969547] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1140.971172] LustreError: 34946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1141.406105] Lustre: server umount lustre-MDT0001 complete [ 1154.344577] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 1157.171401] Key type lgssc unregistered [ 1157.462653] LNet: 38524:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1157.472625] LNetError: 38524:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1157.488676] LNet: Removed LNI 192.168.202.116@tcp [ 1158.492667] Key type .llcrypt unregistered [ 1158.495171] Key type ._llcrypt unregistered [ 1183.578711] Key type ._llcrypt registered [ 1183.581313] Key type .llcrypt registered [ 1183.726514] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 1184.803307] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1184.949150] alg: No test for adler32 (adler32-zlib) [ 1186.035977] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1186.380261] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 1188.103936] Key type lgssc registered [ 1189.557269] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1197.995674] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1198.043375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1199.501379] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1204.058473] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1209.828735] LustreError: 39937:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1209.851619] LustreError: 39937:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1213.815962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1214.365980] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1219.416570] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1228.330791] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1234.915587] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1242.538889] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1242.991387] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1245.056566] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1250.193851] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1258.634260] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1267.051465] Lustre: lustre-OST0000: Not available for connect from 192.168.202.16@tcp (stopping) [ 1269.731062] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1269.749075] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1269.751934] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1269.771574] Lustre: Skipped 1 previous similar message [ 1272.136208] Lustre: lustre-OST0000: Not available for connect from 192.168.202.16@tcp (stopping) [ 1272.148342] Lustre: Skipped 1 previous similar message [ 1272.437639] Lustre: server umount lustre-OST0000 complete [ 1295.329071] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1295.333230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1295.354845] Lustre: Skipped 1 previous similar message [ 1295.368086] Lustre: Skipped 1 previous similar message [ 1299.694733] Lustre: server umount lustre-MDT0000 complete [ 1300.456658] LustreError: 40665:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1300.490365] LustreError: 40665:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1303.906718] LustreError: 39919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442156 with bad export cookie 18321579694281503306 [ 1303.916019] LustreError: 39919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1303.917351] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1318.367537] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1318.563196] Lustre: server umount lustre-MDT0001 complete [ 1330.119821] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 1333.086412] Key type lgssc unregistered [ 1333.383177] LNet: 43652:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1333.405382] LNetError: 43652:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1334.466725] LNet: Removed LNI 192.168.202.116@tcp [ 1335.263817] Key type .llcrypt unregistered [ 1335.266182] Key type ._llcrypt unregistered [ 1360.254587] Key type ._llcrypt registered [ 1360.259102] Key type .llcrypt registered [ 1360.384554] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 1361.548925] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1361.762765] alg: No test for adler32 (adler32-zlib) [ 1362.872717] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1363.083843] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 1364.807226] Key type lgssc registered [ 1366.070923] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1374.966899] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1374.997144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1376.570366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1381.868757] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1386.984732] LustreError: 45086:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1387.016617] LustreError: 45086:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1391.397138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1391.843128] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1396.494552] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1405.620162] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1413.348442] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1420.567791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1421.077668] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1422.153622] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1426.853332] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1435.257257] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1443.180095] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.16@tcp (stopping) [ 1443.297439] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1443.318581] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1446.882827] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1446.883380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1446.893385] Lustre: Skipped 1 previous similar message [ 1446.905219] Lustre: Skipped 3 previous similar messages [ 1448.265376] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.16@tcp (stopping) [ 1448.814464] Lustre: server umount lustre-MDT0000 complete [ 1452.008524] LustreError: 45086:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1452.045319] LustreError: 45086:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1453.249683] LustreError: 45068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442306 with bad export cookie 10201700404450290021 [ 1453.254807] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1453.259686] LustreError: 45068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1453.388241] LustreError: 45083:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1453.437324] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.16@tcp (stopping) [ 1457.121832] LustreError: 45081:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1457.122676] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1458.146608] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1458.151581] Lustre: Skipped 1 previous similar message [ 1459.755599] Lustre: server umount lustre-MDT0001 complete [ 1489.377499] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1489.621240] Lustre: server umount lustre-OST0000 complete [ 1508.587443] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 1511.345167] Key type lgssc unregistered [ 1511.570562] LNet: 48950:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1511.579129] LNetError: 48950:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1511.596247] LNet: Removed LNI 192.168.202.116@tcp [ 1512.486520] Key type .llcrypt unregistered [ 1512.488758] Key type ._llcrypt unregistered [ 1541.411772] Key type ._llcrypt registered [ 1541.415681] Key type .llcrypt registered [ 1541.579399] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 1543.056309] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1543.196613] alg: No test for adler32 (adler32-zlib) [ 1544.195258] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1544.392785] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 1546.119825] Key type lgssc registered [ 1547.430182] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1555.376335] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1555.400400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1570.976125] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1576.095869] LustreError: 50339:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.116@tcp: failed processing log, type 4: rc = -110 [ 1602.784124] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1610.638944] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1618.996458] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1785.628139] Lustre: server umount lustre-OST0000 complete [ 1803.918289] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 1807.072758] Key type lgssc unregistered [ 1807.316872] LNet: 52018:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1807.329448] LNetError: 52018:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1807.345332] LNet: Removed LNI 192.168.202.116@tcp [ 1808.227915] Key type .llcrypt unregistered [ 1808.232489] Key type ._llcrypt unregistered [ 1832.155341] Key type ._llcrypt registered [ 1832.159584] Key type .llcrypt registered [ 1832.335188] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 1834.099152] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1834.134484] alg: No test for adler32 (adler32-zlib) [ 1835.206432] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1835.478029] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 1837.263247] Key type lgssc registered [ 1838.705346] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1848.358226] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1848.404565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1850.219735] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1856.070349] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1859.560301] LustreError: 53435:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1859.596358] LustreError: 53435:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1864.672203] LustreError: 53436:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1865.340434] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1865.700868] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1871.035378] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1880.234763] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1887.563849] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1894.229102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1894.761158] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1902.236824] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1911.112344] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1916.401234] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1916.412868] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1916.466313] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1919.537695] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1919.862315] Lustre: server umount lustre-OST0000 complete [ 1924.583072] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1924.588777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1924.602175] Lustre: Skipped 1 previous similar message [ 1924.621587] Lustre: Skipped 2 previous similar messages [ 1929.702783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1929.711740] Lustre: Skipped 1 previous similar message [ 1930.628291] Lustre: server umount lustre-MDT0000 complete [ 1934.443805] LustreError: 53415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442787 with bad export cookie 10839127538193808585 [ 1934.456317] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1934.460503] LustreError: 53415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1934.800508] Lustre: server umount lustre-MDT0001 complete [ 1946.625612] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 1949.288810] Key type lgssc unregistered [ 1949.647689] LNet: 56972:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1949.659379] LNetError: 56972:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1949.694369] LNet: Removed LNI 192.168.202.116@tcp [ 1950.637782] Key type .llcrypt unregistered [ 1950.639979] Key type ._llcrypt unregistered [ 1974.390739] Key type ._llcrypt registered [ 1974.392409] Key type .llcrypt registered [ 1974.508764] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 1975.343537] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1975.503495] alg: No test for adler32 (adler32-zlib) [ 1976.608908] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1976.855842] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 1978.528750] Key type lgssc registered [ 1979.865987] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1987.568878] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1987.593473] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2003.231545] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2008.351620] LustreError: 58360:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.116@tcp: failed processing log, type 4: rc = -110 [ 2035.039256] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2040.955601] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2048.432169] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2056.583502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2057.165690] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2057.269969] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 2062.151416] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2062.315407] LustreError: 58386:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2062.353175] LustreError: 58386:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2067.432972] LustreError: 58385:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2069.487854] LustreError: 59396:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2069.517565] LustreError: 59396:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2072.560636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2073.105816] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2078.750803] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2087.029225] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2093.897241] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2098.144862] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2098.148158] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2098.166564] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2098.661305] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2098.682654] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2098.698560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2102.497941] Lustre: server umount lustre-OST0000 complete [ 2129.379221] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2129.386922] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2129.392320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2129.399090] Lustre: Skipped 1 previous similar message [ 2130.742387] Lustre: server umount lustre-MDT0000 complete [ 2134.511532] LustreError: 59401:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2135.303727] LustreError: 58366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442988 with bad export cookie 6526719006231111080 [ 2135.308662] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2135.313470] LustreError: 58366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2141.807339] Lustre: server umount lustre-MDT0001 complete [ 2152.256293] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 2155.588506] Key type lgssc unregistered [ 2155.951764] LNet: 62106:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2155.958825] LNetError: 62106:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2155.974405] LNet: Removed LNI 192.168.202.116@tcp [ 2156.838433] Key type .llcrypt unregistered [ 2156.843802] Key type ._llcrypt unregistered [ 2186.981449] Key type ._llcrypt registered [ 2186.983953] Key type .llcrypt registered [ 2187.068522] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 2188.281248] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2188.416316] alg: No test for adler32 (adler32-zlib) [ 2189.734653] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2190.166340] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 2192.032907] Key type lgssc registered [ 2193.488397] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2202.148370] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2202.197955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2203.685455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2207.982742] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2213.857452] LustreError: 63523:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2213.878126] LustreError: 63523:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2217.549377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2218.027775] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2223.853561] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2232.396922] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2239.516165] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2245.439545] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2245.989114] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2250.131750] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2253.476942] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2261.657589] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2280.935491] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2280.946369] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2280.964337] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2284.005242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2284.008354] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2284.020797] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2286.512192] Lustre: server umount lustre-OST0000 complete [ 2292.206823] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2292.212743] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2292.226852] Lustre: Skipped 1 previous similar message [ 2292.238661] Lustre: Skipped 2 previous similar messages [ 2297.314030] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2297.322765] Lustre: Skipped 1 previous similar message [ 2297.603204] Lustre: server umount lustre-MDT0000 complete [ 2301.413911] LustreError: 64252:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2301.458305] LustreError: 64252:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2303.293956] LustreError: 63504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443156 with bad export cookie 8711415885975370680 [ 2303.322184] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2303.841942] Lustre: server umount lustre-MDT0001 complete [ 2316.788587] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 2320.144372] Key type lgssc unregistered [ 2320.508690] LNet: 67081:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2320.513689] LNetError: 67081:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2320.527444] LNet: Removed LNI 192.168.202.116@tcp [ 2321.497493] Key type .llcrypt unregistered [ 2321.503762] Key type ._llcrypt unregistered [ 2356.735636] Key type ._llcrypt registered [ 2356.737986] Key type .llcrypt registered [ 2356.852632] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 2358.439485] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2358.761556] alg: No test for adler32 (adler32-zlib) [ 2359.839121] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2360.120905] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 2361.807845] Key type lgssc registered [ 2363.009493] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2372.861324] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 09:47:03 (1788443223) [ 2380.041125] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2380.073403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2381.590248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2387.100167] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2392.037601] LustreError: 68788:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2392.065736] LustreError: 68788:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2397.153047] LustreError: 68789:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2397.487408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2397.962777] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2403.443773] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2412.172252] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2419.415234] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2426.569727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2426.799427] LustreError: 70610:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2426.981289] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2431.995696] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2433.454749] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2442.202360] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2448.328509] Lustre: Failing over lustre-MDT0000 [ 2448.807914] Lustre: server umount lustre-MDT0000 complete [ 2449.251723] LustreError: 68785:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2449.381285] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2449.387786] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2452.456481] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2454.347917] LustreError: 70393:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2454.384020] LustreError: 70393:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2461.461040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2461.597048] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2461.858743] Lustre: *** cfs_fail_loc=135, val=0*** [ 2462.009371] LustreError: 71875:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2462.019501] LustreError: 71875:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.116@tcp: cfg command failed: rc = -2 [ 2462.028212] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2462.028212] [ 2462.034301] LustreError: MGC192.168.202.116@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2462.054497] LustreError: 71863:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2462.063398] LustreError: 71863:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2462.076172] LustreError: 71863:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2462.082743] LustreError: 71863:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2462.113419] Lustre: server umount lustre-MDT0000 complete [ 2462.119675] LustreError: 71863:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2462.723861] LustreError: 68788:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2462.765491] LustreError: 68788:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 2470.809178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2471.072846] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2471.089583] LustreError: 68780:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2471.423595] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2471.520238] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2474.830163] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2476.396771] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2476.534589] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2476.626512] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 2476.681116] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2486.761125] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2486.766703] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2486.781609] Lustre: Skipped 2 previous similar messages [ 2486.793884] Lustre: Skipped 1 previous similar message [ 2491.421520] Lustre: server umount lustre-OST0000 complete [ 2496.994834] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2497.004313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2502.114079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2502.117265] Lustre: Skipped 2 previous similar messages [ 2506.210836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2506.232254] Lustre: Skipped 1 previous similar message [ 2510.303482] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2510.572752] Lustre: server umount lustre-MDT0000 complete [ 2511.332236] LustreError: 68788:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2511.367895] LustreError: 68788:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 2514.337507] LustreError: 68769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443367 with bad export cookie 2265764175058387426 [ 2514.349177] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2514.360650] LustreError: 68769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2514.891855] Lustre: server umount lustre-MDT0001 complete [ 2526.298329] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 2529.462270] Key type lgssc unregistered [ 2529.760791] LNet: 73662:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2529.770933] LNetError: 73662:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2529.785243] LNet: Removed LNI 192.168.202.116@tcp [ 2530.804562] Key type .llcrypt unregistered [ 2530.808803] Key type ._llcrypt unregistered [ 2559.295460] Key type ._llcrypt registered [ 2559.300481] Key type .llcrypt registered [ 2559.403538] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 2560.685874] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2560.718759] alg: No test for adler32 (adler32-zlib) [ 2561.843520] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2562.230209] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 2563.967198] Key type lgssc registered [ 2565.610935] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2574.697759] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2574.734406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2576.285229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2581.421764] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2586.592196] LustreError: 75073:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2586.616918] LustreError: 75073:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2591.641482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2591.714473] LustreError: 75074:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2592.225723] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2598.422860] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2607.132233] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2613.922953] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2621.388175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2621.690251] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2622.734686] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2629.319295] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2638.142576] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2645.098429] Lustre: Failing over lustre-MDT0000 [ 2645.366554] LustreError: 75068:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2645.461541] Lustre: server umount lustre-MDT0000 complete [ 2647.525339] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2647.530503] LustreError: 75070:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2647.544968] Lustre: Skipped 1 previous similar message [ 2647.566972] LustreError: 75070:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2652.647779] LustreError: 75069:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2652.694586] LustreError: 75069:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2655.587603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2655.819912] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2656.063945] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2656.215641] LustreError: 78161:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2656.227651] LustreError: 78161:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.116@tcp: cfg command failed: rc = -9 [ 2656.238283] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2656.238283] [ 2656.247496] LustreError: MGC192.168.202.116@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2656.259787] LustreError: 78150:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2656.270827] LustreError: 78150:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 2656.291270] LustreError: 78150:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2656.297390] LustreError: 78150:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2656.333769] Lustre: server umount lustre-MDT0000 complete [ 2656.342372] LustreError: 78150:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2660.714863] LustreError: 77407:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2660.728885] LustreError: 77407:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2665.219101] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2665.538298] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2665.553252] LustreError: 75065:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2665.559035] LustreError: 74383:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000007375cd5d x1875318842360320/t0(0) o250->MGC192.168.202.116@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2665.803938] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.16@tcp (not set up) [ 2665.920823] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2665.971971] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2670.942074] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2671.106320] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2671.153499] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2671.210817] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2672.134767] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2681.314212] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2681.342639] Lustre: Skipped 2 previous similar messages [ 2681.346605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2681.364436] Lustre: Skipped 1 previous similar message [ 2686.437805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2686.451390] Lustre: Skipped 1 previous similar message [ 2686.709433] Lustre: server umount lustre-OST0000 complete [ 2691.555392] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2691.559360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2691.573220] Lustre: Skipped 1 previous similar message [ 2691.588521] Lustre: Skipped 1 previous similar message [ 2696.673102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2696.680443] Lustre: Skipped 1 previous similar message [ 2705.376035] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2705.608125] Lustre: server umount lustre-MDT0000 complete [ 2706.912599] LustreError: 77407:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2706.934414] LustreError: 77407:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 2710.254567] LustreError: 75054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443563 with bad export cookie 2032242486030891549 [ 2710.264796] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2710.566527] Lustre: server umount lustre-MDT0001 complete [ 2724.988651] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 2728.558363] Key type lgssc unregistered [ 2729.145973] LNet: 79949:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2729.157805] LNetError: 79949:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2729.184204] LNet: Removed LNI 192.168.202.116@tcp [ 2730.800662] Key type .llcrypt unregistered [ 2730.802488] Key type ._llcrypt unregistered [ 2746.797624] Key type ._llcrypt registered [ 2746.799973] Key type .llcrypt registered [ 2747.000552] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 2747.464356] Key type .llcrypt unregistered [ 2747.471636] Key type ._llcrypt unregistered [ 2772.802607] Key type ._llcrypt registered [ 2772.805607] Key type .llcrypt registered [ 2772.918634] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 2774.073286] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2774.371596] alg: No test for adler32 (adler32-zlib) [ 2775.750729] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2776.057969] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 2777.791499] Key type lgssc registered [ 2779.835809] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2789.355317] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2789.392720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2790.862358] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2796.238953] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2801.128118] LustreError: 81859:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2801.166941] LustreError: 81859:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2804.680363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2804.971540] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2809.499844] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2816.753512] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2822.978096] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2829.625318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2829.737655] LustreError: 83679:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2829.939341] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2836.480653] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2837.468401] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2844.934851] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2855.908090] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2855.914077] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2855.920490] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2856.943606] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2856.944223] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2856.945161] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2856.946892] Lustre: Skipped 1 previous similar message [ 2860.796255] Lustre: server umount lustre-OST0000 complete [ 2866.660683] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2866.678647] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2866.698496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2871.858126] Lustre: server umount lustre-MDT0000 complete [ 2872.303375] LustreError: 81854:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2872.329068] LustreError: 81854:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2877.409600] LustreError: 82888:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2877.449383] LustreError: 82888:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2877.880436] LustreError: 81839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443730 with bad export cookie 12612889241113894337 [ 2877.883101] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2877.901961] LustreError: 81839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2878.848533] Lustre: server umount lustre-MDT0001 complete [ 2896.697667] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 2901.170233] Key type lgssc unregistered [ 2901.837186] LNet: 85421:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2901.864505] LNetError: 85421:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2902.887377] LNet: Removed LNI 192.168.202.116@tcp [ 2904.152578] Key type .llcrypt unregistered [ 2904.156114] Key type ._llcrypt unregistered [ 2931.818491] Key type ._llcrypt registered [ 2931.820500] Key type .llcrypt registered [ 2931.920904] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 2933.124525] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2933.307252] alg: No test for adler32 (adler32-zlib) [ 2934.512649] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2934.864944] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 2936.599901] Key type lgssc registered [ 2937.898175] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2946.186371] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2946.226081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2947.806282] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2953.341775] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2957.282298] LustreError: 86833:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2957.312375] LustreError: 86833:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2962.405941] LustreError: 86832:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2963.494824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2964.224506] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2970.976455] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2982.531798] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2989.517896] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2998.794592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2999.519739] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3007.941524] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3017.023461] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3017.916834] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 3028.456840] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3028.467304] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3028.478109] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3028.504270] Lustre: Skipped 1 previous similar message [ 3033.673858] LustreError: 88656:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3033.829533] Lustre: server umount lustre-OST0000 complete [ 3038.702388] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3038.729432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3038.740051] Lustre: Skipped 2 previous similar messages [ 3043.808558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3043.818715] Lustre: Skipped 1 previous similar message [ 3044.480628] Lustre: server umount lustre-MDT0000 complete [ 3048.340849] LustreError: 86814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443901 with bad export cookie 10204802110517783505 [ 3048.355255] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3048.356558] LustreError: 86814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3048.814823] Lustre: server umount lustre-MDT0001 complete [ 3062.683396] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 3067.443624] Key type lgssc unregistered [ 3067.834466] LNet: 90370:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3067.855153] LNetError: 90370:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3067.868625] LNet: Removed LNI 192.168.202.116@tcp [ 3068.983343] Key type .llcrypt unregistered [ 3068.985200] Key type ._llcrypt unregistered [ 3100.498562] Key type ._llcrypt registered [ 3100.502756] Key type .llcrypt registered [ 3100.636980] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 3101.859653] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3101.905364] alg: No test for adler32 (adler32-zlib) [ 3103.235644] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3103.537118] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 3105.247276] Key type lgssc registered [ 3106.581067] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3115.832846] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3115.845259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3117.340937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3122.419250] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3127.795885] LustreError: 91787:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3127.831212] LustreError: 91787:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3132.910262] LustreError: 91788:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3133.408959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3134.050887] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3139.874480] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3149.134565] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3155.464659] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3162.951569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3163.279528] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3169.272860] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3177.318845] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3187.910768] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 3203.580540] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3203.597434] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3203.597565] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3203.622210] Lustre: Skipped 1 previous similar message [ 3203.655894] Lustre: Skipped 1 previous similar message [ 3207.651613] Lustre: server umount lustre-OST0000 complete [ 3213.792233] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3213.803198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3213.816381] Lustre: Skipped 1 previous similar message [ 3213.835847] Lustre: Skipped 1 previous similar message [ 3218.914876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3218.940865] Lustre: Skipped 1 previous similar message [ 3220.154380] Lustre: server umount lustre-MDT0000 complete [ 3224.034062] LustreError: 91788:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3224.058983] LustreError: 91788:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3224.686542] LustreError: 91767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444077 with bad export cookie 11901144336125831361 [ 3224.697255] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3224.710440] LustreError: 91767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3225.102885] Lustre: server umount lustre-MDT0001 complete [ 3239.488685] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 3242.406285] Key type lgssc unregistered [ 3242.826107] LNet: 95362:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3242.842349] LNetError: 95362:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3242.861736] LNet: Removed LNI 192.168.202.116@tcp [ 3243.921497] Key type .llcrypt unregistered [ 3243.923970] Key type ._llcrypt unregistered [ 3273.040535] Key type ._llcrypt registered [ 3273.043355] Key type .llcrypt registered [ 3273.154735] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 3274.163408] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3274.281881] alg: No test for adler32 (adler32-zlib) [ 3275.298789] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3275.505111] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 3277.143226] Key type lgssc registered [ 3278.406609] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3286.538758] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3286.568377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3302.111601] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3307.231531] LustreError: 96751:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.116@tcp: failed processing log, type 4: rc = -110 [ 3333.919332] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3341.122610] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3349.333972] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3355.331056] Lustre: server umount lustre-OST0000 complete [ 3364.875113] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 10:03:35 (1788444215) [ 3372.515117] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3373.173449] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3378.145491] LustreError: 98390:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3378.171172] LustreError: 98390:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3378.220636] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3382.240711] LustreError: 98391:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3387.362924] LustreError: 98390:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3389.585652] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3390.227038] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3395.929807] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3402.912489] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3409.091666] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3415.880415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3416.249104] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3417.281774] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 3423.738531] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3433.627860] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3448.288762] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3448.296113] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3448.310333] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3450.341290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3450.344247] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3450.370821] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3453.232465] Lustre: server umount lustre-OST0000 complete [ 3460.583260] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3460.592085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3460.604813] Lustre: Skipped 1 previous similar message [ 3460.607607] Lustre: Skipped 1 previous similar message [ 3465.697553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3465.702106] Lustre: Skipped 2 previous similar messages [ 3470.817192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3470.826884] Lustre: Skipped 1 previous similar message [ 3472.351132] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3472.582510] Lustre: server umount lustre-MDT0000 complete [ 3475.938821] LustreError: 98386:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3475.961523] LustreError: 98386:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3476.717040] LustreError: 98370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444329 with bad export cookie 2024985828761440119 [ 3476.721522] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3476.730611] LustreError: 98370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3477.019854] Lustre: server umount lustre-MDT0001 complete [ 3491.737923] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 3494.676172] Key type lgssc unregistered [ 3495.118168] LNet: 102085:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3495.132493] LNetError: 102085:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3495.172726] LNet: Removed LNI 192.168.202.116@tcp [ 3496.333305] Key type .llcrypt unregistered [ 3496.335395] Key type ._llcrypt unregistered [ 3546.570722] Key type ._llcrypt registered [ 3546.587090] Key type .llcrypt registered [ 3546.737711] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 3548.159852] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3548.247748] alg: No test for adler32 (adler32-zlib) [ 3549.385223] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3549.586182] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 3551.287165] Key type lgssc registered [ 3552.450679] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3560.421205] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3560.482072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3562.023455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3566.523587] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3572.196210] LustreError: 104279:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3572.212767] LustreError: 104279:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3574.548167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3574.797604] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3578.687451] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3586.138201] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3592.477956] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3599.515430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3599.825927] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3606.442510] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3607.030153] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 3616.205936] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3629.538894] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3629.546449] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3629.563983] Lustre: Skipped 1 previous similar message [ 3629.574497] Lustre: Skipped 1 previous similar message [ 3634.658078] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3634.665373] Lustre: Skipped 1 previous similar message [ 3635.599519] Lustre: server umount lustre-OST0000 complete [ 3639.776331] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3639.783085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3639.785105] Lustre: Skipped 1 previous similar message [ 3639.812328] Lustre: Skipped 1 previous similar message [ 3644.900640] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3645.777959] Lustre: server umount lustre-MDT0000 complete [ 3649.095715] LustreError: 104257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444501 with bad export cookie 3262416880600704584 [ 3649.101962] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3649.107597] LustreError: 104257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3649.428307] Lustre: server umount lustre-MDT0001 complete [ 3660.101244] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 3663.564234] Key type lgssc unregistered [ 3663.840707] LNet: 107834:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3663.853619] LNetError: 107834:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3663.867618] LNet: Removed LNI 192.168.202.116@tcp [ 3664.777204] Key type .llcrypt unregistered [ 3664.779182] Key type ._llcrypt unregistered [ 3682.661390] Key type ._llcrypt registered [ 3682.664824] Key type .llcrypt registered [ 3682.773723] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 3684.065848] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3684.112335] alg: No test for adler32 (adler32-zlib) [ 3685.210989] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3685.489914] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 3687.215168] Key type lgssc registered [ 3688.158465] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3696.561217] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3696.591587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3712.160097] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3717.279424] LustreError: 109076:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.116@tcp: failed processing log, type 4: rc = -110 [ 3743.903690] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3749.030692] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3757.305602] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3764.098989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3764.415572] LustreError: MGC192.168.202.116@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3764.428385] LustreError: 110100:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3764.467197] LustreError: 110100:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3764.477223] LustreError: 110100:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3764.645364] Lustre: server umount lustre-MDT0000 complete [ 3764.647397] LustreError: 110100:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3785.695172] Lustre: 108413:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788444622/real 1788444622] req@000000002e90c952 x1875320019881216/t0(0) o400->MGC192.168.202.116@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788444638 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3785.729955] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3785.749598] LustreError: 109088:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3806.306607] Lustre: server umount lustre-OST0000 complete [ 3815.332640] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 3824.987154] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 3866.096131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3866.496052] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3866.549591] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3866.728511] Lustre: lustre-MDT0000: new disk, initializing [ 3866.831499] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3866.866879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3871.164581] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3881.661927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3881.762689] Lustre: 112897:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3881.822600] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3881.825531] Lustre: Skipped 1 previous similar message [ 3881.896657] Lustre: lustre-MDT0001: new disk, initializing [ 3882.014520] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3882.074917] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3882.098843] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3886.566925] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3896.771595] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3902.372751] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3911.284052] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3911.822848] Lustre: lustre-OST0000: new disk, initializing [ 3911.841319] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3911.862458] Lustre: 114866:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3912.059055] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3919.597410] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3919.623244] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3919.831627] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3921.410759] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3933.644843] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3938.215848] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3938.343799] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3944.307230] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3944.709679] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3950.564153] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3950.566654] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3950.599674] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3950.618697] Lustre: Skipped 1 previous similar message [ 3955.692862] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3955.706541] Lustre: Skipped 1 previous similar message [ 3960.804721] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3960.811873] Lustre: Skipped 1 previous similar message [ 3961.311361] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3961.482200] Lustre: server umount lustre-OST0000 complete [ 3965.921910] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3965.947065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3965.952949] Lustre: Skipped 2 previous similar messages [ 3965.974790] Lustre: Skipped 1 previous similar message [ 3971.042107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3971.050659] Lustre: Skipped 1 previous similar message [ 3972.019833] Lustre: server umount lustre-MDT0000 complete [ 3975.937754] LustreError: 112890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444828 with bad export cookie 15405048889863590987 [ 3975.953164] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3975.962317] LustreError: 112890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3976.557813] Lustre: server umount lustre-MDT0001 complete [ 3986.579215] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 10:13:57 (1788444837) [ 3990.964425] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 4009.777604] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 4022.902421] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 4059.620264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4059.958696] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4060.050583] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4060.107269] Lustre: lustre-MDT0000: new disk, initializing [ 4060.146839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4060.155788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4065.922676] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4078.570539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4078.728479] Lustre: 119771:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4078.773576] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4078.777971] Lustre: Skipped 1 previous similar message [ 4078.872631] Lustre: lustre-MDT0001: new disk, initializing [ 4079.016685] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4079.058021] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4079.063582] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4085.201106] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4100.397905] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4107.494459] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4116.470707] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4116.877461] Lustre: lustre-OST0000: new disk, initializing [ 4116.901856] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4116.916452] Lustre: 121740:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4117.080438] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4124.268438] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4124.283073] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4124.421438] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4124.603845] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4136.119098] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4141.957590] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4142.291136] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4147.685620] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4147.929072] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4155.367075] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4155.370187] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4155.390721] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4155.394394] LustreError: Skipped 1 previous similar message [ 4156.955825] Lustre: server umount lustre-OST0000 complete [ 4160.993987] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4161.009307] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4161.028600] Lustre: Skipped 1 previous similar message [ 4165.617056] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4166.906691] Lustre: server umount lustre-MDT0000 complete [ 4171.346550] LustreError: 119763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445024 with bad export cookie 15405048889863591869 [ 4171.350343] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4171.357192] LustreError: 119763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4171.714838] Lustre: server umount lustre-MDT0001 complete [ 4182.034167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4182.645233] LustreError: 123961:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4182.666191] LustreError: 123961:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4182.732591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4187.503892] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4188.133803] LustreError: 123962:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4192.260961] LustreError: 123961:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4197.350767] LustreError: 123962:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4198.734665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4204.802841] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4212.879975] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4219.461094] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4226.661478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4233.002416] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4240.426977] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4250.391659] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 4259.296838] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4259.306384] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4259.306630] LustreError: Skipped 1 previous similar message [ 4259.309726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4259.320142] Lustre: Skipped 1 previous similar message [ 4259.350237] Lustre: Skipped 5 previous similar messages [ 4261.896639] Lustre: server umount lustre-OST0000 complete [ 4272.949929] Lustre: server umount lustre-MDT0000 complete [ 4274.657369] LustreError: 123957:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4274.682573] LustreError: 123957:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4277.828870] LustreError: 126842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445130 with bad export cookie 15405048889863592744 [ 4277.830638] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4277.841132] LustreError: 126842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4278.360323] Lustre: server umount lustre-MDT0001 complete [ 4296.193870] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 4299.837834] Key type lgssc unregistered [ 4300.138624] LNet: 127598:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4300.143315] LNetError: 127598:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4300.163913] LNet: Removed LNI 192.168.202.116@tcp [ 4301.326109] Key type .llcrypt unregistered [ 4301.328198] Key type ._llcrypt unregistered [ 4321.450951] Key type ._llcrypt registered [ 4321.453043] Key type .llcrypt registered [ 4321.530680] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 4337.868251] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 4338.903682] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4338.942075] alg: No test for adler32 (adler32-zlib) [ 4340.048198] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 4340.272405] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 4341.943262] Key type lgssc registered [ 4343.411807] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4383.192467] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4383.240371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4384.761881] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4384.839050] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4384.916308] Lustre: lustre-MDT0000: new disk, initializing [ 4385.022075] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4385.039534] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4389.957617] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4404.810509] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4405.145710] Lustre: 130762:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4405.278963] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4405.289479] Lustre: Skipped 1 previous similar message [ 4405.501708] Lustre: lustre-MDT0001: new disk, initializing [ 4405.634781] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4405.705690] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4405.752236] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4411.044492] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4422.544907] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4430.009971] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4438.771631] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4439.152234] Lustre: lustre-OST0000: new disk, initializing [ 4439.155687] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4439.163382] Lustre: 132731:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4439.248325] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4445.195448] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4445.773798] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4445.786067] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4445.917427] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4456.984509] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4462.396240] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4462.591823] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4467.176681] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4467.487571] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4471.788296] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4471.799065] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4471.810764] Lustre: Skipped 1 previous similar message [ 4471.811176] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4471.821081] Lustre: Skipped 1 previous similar message [ 4475.235180] Lustre: server umount lustre-OST0000 complete [ 4482.024379] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4482.030476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4482.035875] Lustre: Skipped 1 previous similar message [ 4482.059703] Lustre: Skipped 1 previous similar message [ 4487.137087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4487.145228] Lustre: Skipped 1 previous similar message [ 4491.235313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4491.243714] Lustre: Skipped 1 previous similar message [ 4493.279302] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4493.870316] Lustre: server umount lustre-MDT0000 complete [ 4498.122027] LustreError: 132730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445350 with bad export cookie 10527207528923036375 [ 4498.130436] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4498.139690] LustreError: 132730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4498.558258] Lustre: server umount lustre-MDT0001 complete [ 4509.141677] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 10:22:39 (1788445359) [ 4517.406091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4518.217124] LustreError: 135147:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4518.242955] LustreError: 135147:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4518.357211] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4523.467858] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4523.491644] LustreError: 135148:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4528.608832] LustreError: 135147:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4533.625423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4533.737963] LustreError: 135148:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4534.022205] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4538.748078] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4547.320508] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4554.311880] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4558.374393] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4558.377653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4558.396088] Lustre: Skipped 1 previous similar message [ 4558.414851] Lustre: Skipped 1 previous similar message [ 4562.858526] Lustre: server umount lustre-MDT0000 complete [ 4563.424121] LustreError: 135147:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4563.445943] LustreError: 135147:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4566.363421] LustreError: 135127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445419 with bad export cookie 10527207528923037257 [ 4566.374697] LustreError: 135127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4566.391633] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4566.731516] Lustre: server umount lustre-MDT0001 complete [ 4576.827053] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 10:23:47 (1788445427) [ 4584.025277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4599.712817] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4604.831737] LustreError: 137563:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.116@tcp: failed processing log, type 4: rc = -110 [ 4630.495994] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4637.962901] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4647.210201] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4649.571674] Lustre: server umount lustre-OST0000 complete [ 4661.500147] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 10:25:11 (1788445511) [ 4671.500228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4672.097656] LustreError: 139009:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4672.155974] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4676.926445] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4686.137224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4691.172345] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4699.949625] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4706.979272] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4715.050087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4715.476202] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4715.488935] Lustre: Skipped 1 previous similar message [ 4722.446355] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4731.167245] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4742.114218] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4742.129155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4742.131394] Lustre: Skipped 1 previous similar message [ 4742.159338] Lustre: Skipped 2 previous similar messages [ 4746.680140] Lustre: server umount lustre-MDT0000 complete [ 4747.249144] LustreError: 140026:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4747.298667] LustreError: 140026:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 4751.603766] LustreError: 140854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445604 with bad export cookie 10527207528923037929 [ 4751.613402] LustreError: 140854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4751.625574] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4752.179354] Lustre: server umount lustre-MDT0001 complete [ 4766.955644] Lustre: server umount lustre-OST0000 complete [ 4777.138025] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 10:27:07 (1788445627) [ 4786.118724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4786.589330] LustreError: 142669:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4786.649401] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4792.744186] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4802.505267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4807.368932] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4816.085423] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4822.329873] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4829.157126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4831.682026] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4835.922781] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4845.873425] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4851.013970] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4851.245245] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4856.378402] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4856.606059] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4862.439779] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4862.449591] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4862.466335] Lustre: Skipped 1 previous similar message [ 4862.483894] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4863.464950] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4865.492311] Lustre: server umount lustre-OST0000 complete [ 4869.599945] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4873.699533] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4873.723211] Lustre: Skipped 2 previous similar messages [ 4875.824934] Lustre: server umount lustre-MDT0000 complete [ 4878.818605] LustreError: 142665:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4878.838486] LustreError: 142665:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 4880.196183] LustreError: 142651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445732 with bad export cookie 10527207528923039000 [ 4880.206935] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4880.215335] LustreError: 142651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4880.646698] Lustre: server umount lustre-MDT0001 complete [ 4890.177995] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 10:29:01 (1788445741) [ 4897.228358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4912.991842] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4918.111610] LustreError: 146830:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.116@tcp: failed processing log, type 4: rc = -110 [ 4943.839564] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4943.842839] Lustre: Skipped 2 previous similar messages [ 4951.574774] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4963.621502] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4973.470591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4974.067306] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4978.149094] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4986.401730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4990.735552] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4998.438095] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5005.118351] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5010.167515] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5010.463567] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5014.751719] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5014.978956] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5017.575148] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5017.577313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5017.604353] Lustre: Skipped 2 previous similar messages [ 5017.622170] Lustre: Skipped 7 previous similar messages [ 5031.391230] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5031.616975] Lustre: server umount lustre-MDT0000 complete [ 5032.934801] LustreError: 146855:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5032.962498] LustreError: 146855:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 5035.775908] LustreError: 146837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445888 with bad export cookie 10527207528923040022 [ 5035.781158] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5035.794930] LustreError: 146837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5036.265689] Lustre: server umount lustre-MDT0001 complete [ 5051.100619] Lustre: server umount lustre-OST0000 complete [ 5061.650400] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 10:31:52 (1788445912) [ 5069.928351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5085.664569] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5090.786577] LustreError: 151028:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.116@tcp: failed processing log, type 4: rc = -110 [ 5123.908823] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5133.620236] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5143.816042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5144.923049] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 5151.185359] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5160.768039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5165.646934] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5173.620674] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5179.857557] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5187.060501] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5187.215199] Lustre: lustre-OST0001: new disk, initializing [ 5187.219823] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5187.225928] Lustre: 153902:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5193.388968] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5196.377311] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5196.394549] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5196.514095] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5203.024178] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5207.673826] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5207.898569] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5213.431254] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5213.672137] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5216.224210] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5216.226689] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5216.245818] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5216.249515] Lustre: Skipped 6 previous similar messages [ 5230.048244] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5230.138909] Lustre: server umount lustre-OST0000 complete [ 5237.743331] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5237.779858] LustreError: Skipped 1 previous similar message [ 5241.432807] Lustre: server umount lustre-OST0001 complete [ 5252.870807] Lustre: server umount lustre-MDT0000 complete [ 5257.542587] LustreError: 153901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446110 with bad export cookie 10527207528923040974 [ 5257.555588] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5257.563704] LustreError: 153901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5258.200500] Lustre: server umount lustre-MDT0001 complete [ 5286.533373] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 10:35:37 (1788446137) [ 5288.597273] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 5291.165808] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 5292.916306] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 10:35:44 (1788446144) [ 5300.514815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5300.862781] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5300.912476] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5301.174101] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5301.177972] Lustre: Skipped 6 previous similar messages [ 5305.918297] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5316.127113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5316.189395] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5316.216572] Lustre: 157468:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5320.875494] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5329.023689] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5335.323154] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5342.720768] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5342.962679] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5349.349336] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5350.049050] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 5358.292858] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5363.599707] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5363.848481] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5369.323550] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5369.547308] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5375.456193] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5375.461808] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5375.482986] Lustre: Skipped 5 previous similar messages [ 5375.498408] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5375.507088] Lustre: Skipped 10 previous similar messages [ 5377.863167] Lustre: server umount lustre-OST0000 complete [ 5400.369198] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5406.494443] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5415.095954] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5416.147205] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5416.171683] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5416.173039] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5416.203049] Lustre: Skipped 1 previous similar message [ 5420.443484] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5420.635390] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5425.428970] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5425.602516] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5431.853208] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff960985517000.ost_server_uuid 50 [ 5433.698921] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff960985517000.ost_server_uuid in FULL state after 0 sec [ 5433.927890] LustreError: 157474:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.202.16@tcp) returned error from blocking AST (req@0000000062dd9ff0 x1875320706966016 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 0000000005be9099/0x92182778d6469628 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.16@tcp remote: 0x782fb633bfea66c2 expref: 5 pid: 157473 timeout: 5533 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5433.953092] LustreError: lustre-MDT0000: A client on nid 192.168.202.16@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5433.965366] LustreError: 157463:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.202.16@tcp ns: mdt-lustre-MDT0000_UUID lock: 0000000005be9099/0x92182778d6469628 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.16@tcp remote: 0x782fb633bfea66c2 expref: 6 pid: 157473 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5439.978872] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5439.982449] LustreError: Skipped 1 previous similar message [ 5445.251658] Lustre: server umount lustre-OST0000 complete [ 5455.664732] Lustre: server umount lustre-MDT0000 complete [ 5457.377466] LustreError: 157479:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5457.410422] LustreError: 157479:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 5460.480243] LustreError: 159329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446313 with bad export cookie 10527207528923042157 [ 5460.481620] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5460.500996] LustreError: 159329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5466.977556] Lustre: server umount lustre-MDT0001 complete [ 5478.783637] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 5481.879099] Key type lgssc unregistered [ 5482.333521] LNet: 163587:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5482.344126] LNetError: 163587:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5482.375561] LNet: Removed LNI 192.168.202.116@tcp [ 5483.370832] Key type .llcrypt unregistered [ 5483.372632] Key type ._llcrypt unregistered [ 5511.691538] Key type ._llcrypt registered [ 5511.694130] Key type .llcrypt registered [ 5511.793268] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 5512.793621] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5512.971337] alg: No test for adler32 (adler32-zlib) [ 5514.356446] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 5514.618579] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 5516.311179] Key type lgssc registered [ 5517.363812] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5526.319095] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5526.336396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5527.954575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5534.019861] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5537.259566] LustreError: 165002:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5537.286150] LustreError: 165002:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5542.369113] LustreError: 165003:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5543.106796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5543.572761] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5548.499978] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5557.386681] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5564.232292] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5571.112085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5571.434622] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5572.470169] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5578.738179] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5586.934840] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5593.786555] Lustre: Failing over lustre-MDT0000 [ 5594.086183] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5594.097184] Lustre: Skipped 2 previous similar messages [ 5594.097629] LustreError: 164998:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5594.132655] LustreError: 164998:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5594.285467] Lustre: server umount lustre-MDT0000 complete [ 5599.207031] LustreError: 165730:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5599.220856] LustreError: 165730:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5604.324976] LustreError: 165002:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5604.354429] LustreError: 165002:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5606.806984] LustreError: 164983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.16@tcp arrived at 1788446459 with bad export cookie 1616777509543034411 [ 5609.440525] Lustre: 164310:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788446446/real 1788446446] req@00000000cfb968b7 x1875321937751424/t0(0) o400->MGC192.168.202.116@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788446462 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5609.482310] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5614.560805] LustreError: 166486:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5614.599438] LustreError: 166486:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 5617.038609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5619.681186] LustreError: 164309:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000005c67f62e x1875321937755648/t0(0) o250->MGC192.168.202.116@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5619.959803] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5620.008831] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5625.330304] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5625.341265] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5625.535739] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5628.661373] Lustre: lustre-MDT0000: Denying connection for new client 78912e4d-ace6-4ad7-ba58-40052ec27911 (at 192.168.202.16@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:06 [ 5640.167281] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5640.176294] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5640.191187] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5642.620991] Lustre: server umount lustre-OST0000 complete [ 5646.314251] LustreError: 168733:0:(ldlm_lib.c:3004:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5646.321705] Lustre: 168082:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5646.328989] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5646.343948] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5646.353571] LustreError: 168082:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000b95c944d x1875321937770880/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5646.367091] LustreError: 168082:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5646.375247] LustreError: 168082:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5646.382061] Lustre: lustre-MDT0000: Recovery over after 0:21, of 2 clients 0 recovered and 2 were evicted. [ 5646.750663] Lustre: server umount lustre-MDT0000 complete [ 5650.868868] LustreError: 164983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446503 with bad export cookie 1616777509543034516 [ 5650.880873] LustreError: 164983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5650.891497] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5650.898479] LustreError: 164309:0:(import.c:707:ptlrpc_connect_import_locked()) can't connect to a closed import [ 5653.227576] Lustre: server umount lustre-MDT0001 complete [ 5666.061402] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 5669.427677] Key type lgssc unregistered [ 5669.796916] LNet: 169490:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5669.808436] LNetError: 169490:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5669.825378] LNet: Removed LNI 192.168.202.116@tcp [ 5670.744167] Key type .llcrypt unregistered [ 5670.747556] Key type ._llcrypt unregistered [ 5696.255917] Key type ._llcrypt registered [ 5696.263603] Key type .llcrypt registered [ 5696.422722] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 5697.949228] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5697.976618] alg: No test for adler32 (adler32-zlib) [ 5699.064596] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 5699.276332] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 5700.927171] Key type lgssc registered [ 5701.964778] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5709.940760] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5709.959572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5711.601143] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5716.436129] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5722.080514] LustreError: 170905:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5722.118220] LustreError: 170905:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5725.727585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5726.308378] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5731.545691] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5740.568342] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5747.408059] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5756.478815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5756.945774] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5762.066744] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5763.907536] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5772.352663] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5780.453308] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5780.463479] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5780.482497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5785.243897] Lustre: server umount lustre-OST0000 complete [ 5790.689922] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5790.730426] Lustre: Skipped 1 previous similar message [ 5790.742484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5790.745322] Lustre: Skipped 1 previous similar message [ 5795.155366] Lustre: server umount lustre-MDT0000 complete [ 5795.811591] LustreError: 170901:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5795.834572] LustreError: 170901:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5799.034695] LustreError: 170885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446651 with bad export cookie 7484913456515425186 [ 5799.038154] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5799.041571] LustreError: 170885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5799.354276] Lustre: server umount lustre-MDT0001 complete [ 5811.368912] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 5814.445835] Key type lgssc unregistered [ 5814.851691] LNet: 174442:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5814.858166] LNetError: 174442:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5814.885258] LNet: Removed LNI 192.168.202.116@tcp [ 5816.088294] Key type .llcrypt unregistered [ 5816.094807] Key type ._llcrypt unregistered [ 5853.882804] Key type ._llcrypt registered [ 5853.884819] Key type .llcrypt registered [ 5853.973922] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 5855.511615] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5855.545684] alg: No test for adler32 (adler32-zlib) [ 5856.626772] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 5856.837837] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 5858.503208] Key type lgssc registered [ 5859.649563] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5868.047961] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5868.092193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5869.573861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5874.586311] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5879.782577] LustreError: 176152:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5879.806801] LustreError: 176152:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5884.561216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5884.957390] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5889.558718] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5898.043125] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5903.771718] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5910.322537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5910.755259] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5914.880592] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5916.932189] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5925.482933] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5940.706706] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5940.715639] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5940.726728] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5940.732918] Lustre: Skipped 1 previous similar message [ 5940.741901] Lustre: Skipped 1 previous similar message [ 5945.746786] Lustre: server umount lustre-OST0000 complete [ 5950.949542] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5950.962834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5950.971751] Lustre: Skipped 1 previous similar message [ 5950.981540] Lustre: Skipped 1 previous similar message [ 5955.747878] Lustre: server umount lustre-MDT0000 complete [ 5956.064311] LustreError: 176147:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5956.074748] LustreError: 176147:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5959.255476] LustreError: 177993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446812 with bad export cookie 11863276028631298774 [ 5959.258929] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5959.270216] LustreError: 177993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5959.634817] Lustre: server umount lustre-MDT0001 complete [ 5973.143628] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 5976.504128] Key type lgssc unregistered [ 5976.843634] LNet: 179706:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5976.855880] LNetError: 179706:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5976.868968] LNet: Removed LNI 192.168.202.116@tcp [ 5977.767196] Key type .llcrypt unregistered [ 5977.768927] Key type ._llcrypt unregistered [ 6004.461685] Key type ._llcrypt registered [ 6004.464638] Key type .llcrypt registered [ 6004.562578] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 6005.945484] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6005.965933] alg: No test for adler32 (adler32-zlib) [ 6007.104821] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 6007.289267] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 6008.927698] Key type lgssc registered [ 6009.974493] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6020.537622] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6020.556213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6021.885895] Lustre: *** cfs_fail_loc=135, val=0*** [ 6021.955421] LustreError: 181231:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 6021.960831] LustreError: 181231:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.116@tcp: cfg command failed: rc = -2 [ 6021.967403] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 6021.967403] [ 6021.978430] LustreError: MGC192.168.202.116@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 6021.997480] LustreError: 181192:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 6022.004959] LustreError: 181192:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 6022.014593] LustreError: 181192:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 6022.020619] LustreError: 181192:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 6022.084244] Lustre: server umount lustre-MDT0000 complete [ 6022.086632] LustreError: 181192:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 6028.519329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6028.849258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6033.090847] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6039.018617] LustreError: 181603:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6039.042060] LustreError: 181603:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6041.083643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6041.352333] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6045.279988] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6053.164535] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6059.821092] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6064.096964] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6064.111181] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6064.118325] Lustre: Skipped 1 previous similar message [ 6067.183509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6068.527190] Lustre: server umount lustre-MDT0000 complete [ 6072.290944] LustreError: 181603:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6072.301029] LustreError: 181603:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6072.468862] LustreError: 181585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446925 with bad export cookie 3368187721776195661 [ 6072.487388] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6072.779653] Lustre: server umount lustre-MDT0001 complete [ 6083.679455] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 6086.283861] Key type lgssc unregistered [ 6086.565717] LNet: 183925:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6086.581522] LNetError: 183925:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6087.657046] LNet: Removed LNI 192.168.202.116@tcp [ 6088.319978] Key type .llcrypt unregistered [ 6088.325136] Key type ._llcrypt unregistered [ 6109.608556] Key type ._llcrypt registered [ 6109.611141] Key type .llcrypt registered [ 6109.742268] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 6110.112306] Key type .llcrypt unregistered [ 6110.114156] Key type ._llcrypt unregistered [ 6128.589708] Key type ._llcrypt registered [ 6128.593405] Key type .llcrypt registered [ 6128.729727] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 6129.920458] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6129.988138] alg: No test for adler32 (adler32-zlib) [ 6131.078830] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 6131.330642] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 6133.079424] Key type lgssc registered [ 6134.157588] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6140.812434] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6140.824891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6156.319830] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6161.439333] LustreError: 186061:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.116@tcp: failed processing log, type 4: rc = -110 [ 6188.128082] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6194.780024] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6202.583539] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6210.883385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6211.500989] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6211.551940] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 6216.678591] LustreError: 186087:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6216.701478] LustreError: 186087:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6216.792506] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6221.798742] LustreError: 186086:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6222.829679] LustreError: 187099:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6222.853606] LustreError: 187099:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6225.467928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6225.917518] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6230.771862] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6240.235789] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6247.320419] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6250.286416] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6261.728846] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6261.742416] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6261.805176] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6261.814088] Lustre: Skipped 1 previous similar message [ 6266.848842] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6266.862648] Lustre: Skipped 1 previous similar message [ 6271.975367] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6271.979720] Lustre: Skipped 1 previous similar message [ 6275.040059] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6275.264212] Lustre: server umount lustre-OST0000 complete [ 6282.209484] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6282.213829] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6282.232142] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6282.248920] Lustre: Skipped 2 previous similar messages [ 6282.266287] Lustre: Skipped 1 previous similar message [ 6285.722964] Lustre: server umount lustre-MDT0000 complete [ 6287.329761] LustreError: 187105:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6287.339365] LustreError: 187105:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6289.091801] LustreError: 186067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447141 with bad export cookie 9144119180847550509 [ 6289.096526] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6289.100597] LustreError: 186067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6289.480633] Lustre: server umount lustre-MDT0001 complete [ 6298.927045] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 6302.091210] Key type lgssc unregistered [ 6302.400779] LNet: 189983:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6302.406240] LNetError: 189983:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6302.423327] LNet: Removed LNI 192.168.202.116@tcp [ 6303.445589] Key type .llcrypt unregistered [ 6303.450868] Key type ._llcrypt unregistered [ 6330.143336] Key type ._llcrypt registered [ 6330.145653] Key type .llcrypt registered [ 6330.233824] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 6331.368705] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6331.501845] alg: No test for adler32 (adler32-zlib) [ 6332.660472] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 6332.893124] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 6334.552908] Key type lgssc registered [ 6335.808209] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6344.731504] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6344.746748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6346.475390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6352.305678] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6356.978118] LustreError: 191397:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6357.008857] LustreError: 191397:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6362.086241] LustreError: 191398:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6363.166291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6363.502342] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6367.862768] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6375.811766] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6381.634246] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6387.695545] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6388.100322] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6392.252391] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 6394.017286] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6401.628530] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6405.897330] Lustre: Failing over lustre-MDT0000 [ 6406.135941] Lustre: server umount lustre-MDT0000 complete [ 6407.648867] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6407.663693] LustreError: 191394:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6407.669023] Lustre: Skipped 1 previous similar message [ 6407.679830] LustreError: 191394:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6412.779516] LustreError: 191398:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6412.793643] LustreError: 191398:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 6417.890234] LustreError: 193301:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6417.911147] LustreError: 193301:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6424.037332] Lustre: 190707:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788447260/real 1788447260] req@00000000c96a7677 x1875322795484928/t0(0) o400->MGC192.168.202.116@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788447276 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6424.067556] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6425.920719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6429.064582] LustreError: 191392:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6429.078167] LustreError: 191392:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 6433.249762] LustreError: 190706:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000c726b2a8 x1875322795490432/t0(0) o250->MGC192.168.202.116@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6433.541377] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6433.611766] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6435.146627] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6437.832299] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6438.901213] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6438.921206] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 6438.965448] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 6442.349815] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6452.325413] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6470.112697] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6470.121533] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6470.141847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6471.136548] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6471.149939] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6471.161315] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6472.679986] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6472.695559] Lustre: Skipped 1 previous similar message [ 6474.822797] Lustre: server umount lustre-OST0000 complete [ 6479.846891] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6479.852090] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6479.864563] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6485.330832] Lustre: server umount lustre-MDT0000 complete [ 6488.032593] LustreError: 191398:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6488.045183] LustreError: 191398:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 6489.286073] LustreError: 191378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447342 with bad export cookie 9404440059555005903 [ 6489.293041] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6489.302147] LustreError: 191378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6489.735743] Lustre: server umount lustre-MDT0001 complete [ 6504.954762] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 6507.801803] Key type lgssc unregistered [ 6508.011413] LNet: 196360:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6508.015523] LNetError: 196360:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6508.032964] LNet: Removed LNI 192.168.202.116@tcp [ 6509.445742] Key type .llcrypt unregistered [ 6509.449086] Key type ._llcrypt unregistered [ 6537.106446] Key type ._llcrypt registered [ 6537.108910] Key type .llcrypt registered [ 6537.237996] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 6538.787259] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6539.059124] alg: No test for adler32 (adler32-zlib) [ 6540.214101] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 6540.477528] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 6542.167723] Key type lgssc registered [ 6543.604203] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6551.761158] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6551.787320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6553.533563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6558.726317] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6563.811775] LustreError: 197777:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6563.839418] LustreError: 197777:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6568.962440] LustreError: 197778:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6569.945828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6570.307971] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6576.103588] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6585.167255] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6591.390505] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6599.868717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6600.226519] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6606.964152] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6614.570875] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6623.798222] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6628.573580] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6629.669291] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6649.265341] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6656.484820] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6656.492501] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6656.508589] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6659.566414] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6659.569028] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6659.587708] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6661.035757] Lustre: server umount lustre-OST0000 complete [ 6667.744432] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6667.749296] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6667.783333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6667.786202] Lustre: Skipped 1 previous similar message [ 6669.795026] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6669.798547] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6669.827052] Lustre: Skipped 1 previous similar message [ 6671.128196] Lustre: server umount lustre-MDT0000 complete [ 6674.912111] LustreError: 197777:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6674.932220] LustreError: 197777:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6675.095907] LustreError: 197758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447527 with bad export cookie 5829996305262193130 [ 6675.110798] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6675.113483] LustreError: 197758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6675.405174] Lustre: server umount lustre-MDT0001 complete [ 6688.739153] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 6692.893428] Key type lgssc unregistered [ 6693.266976] LNet: 201671:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6693.278701] LNetError: 201671:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6693.316159] LNet: Removed LNI 192.168.202.116@tcp [ 6694.439746] Key type .llcrypt unregistered [ 6694.441489] Key type ._llcrypt unregistered [ 6721.577508] Key type ._llcrypt registered [ 6721.588978] Key type .llcrypt registered [ 6721.731699] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 6723.391883] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6723.564976] alg: No test for adler32 (adler32-zlib) [ 6724.947491] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 6725.311062] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 6727.047283] Key type lgssc registered [ 6728.154168] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6737.051451] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6737.061386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6738.647320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6744.005390] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6749.154816] LustreError: 202940:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6749.168432] LustreError: 202940:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6754.273450] LustreError: 202939:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6755.501397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6756.289501] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6762.521211] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6770.991826] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6779.588886] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6786.668762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6787.162547] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6794.253372] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6802.552625] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6808.698322] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6814.236557] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6825.997117] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6835.035874] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6844.390847] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6855.138789] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6855.152520] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6855.182628] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6860.257970] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6860.260332] Lustre: Skipped 1 previous similar message [ 6860.941954] Lustre: server umount lustre-OST0000 complete [ 6865.377665] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6865.403444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6865.409797] Lustre: Skipped 2 previous similar messages [ 6865.438621] Lustre: Skipped 2 previous similar messages [ 6870.496494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6870.921617] Lustre: server umount lustre-MDT0000 complete [ 6875.617349] LustreError: 202939:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6875.639518] LustreError: 202939:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6875.817196] LustreError: 205630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447728 with bad export cookie 9048251022222674355 [ 6875.821107] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6875.827521] LustreError: 205630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6876.233610] Lustre: server umount lustre-MDT0001 complete [ 6889.446631] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 6892.129059] Key type lgssc unregistered [ 6892.346817] LNet: 207832:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6892.354763] LNetError: 207832:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6892.379315] LNet: Removed LNI 192.168.202.116@tcp [ 6893.181715] Key type .llcrypt unregistered [ 6893.184307] Key type ._llcrypt unregistered [ 6919.344793] Key type ._llcrypt registered [ 6919.349891] Key type .llcrypt registered [ 6919.473100] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 6920.504333] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6920.532636] alg: No test for adler32 (adler32-zlib) [ 6921.875454] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 6922.056796] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 6923.703191] Key type lgssc registered [ 6925.199459] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6933.912724] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6933.939063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6935.468667] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6940.576859] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6945.769236] LustreError: 209245:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6945.788525] LustreError: 209245:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6949.798115] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6950.097552] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6953.990708] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6962.392591] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6968.863996] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6976.508720] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6976.931735] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6981.044110] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6984.857572] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 6997.177663] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7014.880565] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7014.885418] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7014.886730] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7014.893438] Lustre: Skipped 1 previous similar message [ 7014.900076] Lustre: Skipped 1 previous similar message [ 7020.000783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7020.022917] Lustre: Skipped 1 previous similar message [ 7021.114382] Lustre: server umount lustre-OST0000 complete [ 7027.169196] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7027.174580] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7027.181299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7030.241378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7030.241655] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7031.763735] Lustre: server umount lustre-MDT0000 complete [ 7035.371630] LustreError: 209246:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7035.434525] LustreError: 209246:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7037.539742] LustreError: 209227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447890 with bad export cookie 2982107378362542304 [ 7037.556666] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7037.568667] LustreError: 209227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7038.228905] Lustre: server umount lustre-MDT0001 complete [ 7053.872338] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 7057.489360] Key type lgssc unregistered [ 7057.806719] LNet: 212934:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7057.813872] LNetError: 212934:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7057.831879] LNet: Removed LNI 192.168.202.116@tcp [ 7058.801327] Key type .llcrypt unregistered [ 7058.802814] Key type ._llcrypt unregistered [ 7082.277916] Key type ._llcrypt registered [ 7082.284302] Key type .llcrypt registered [ 7082.434462] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 7083.823144] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7084.048275] alg: No test for adler32 (adler32-zlib) [ 7085.327365] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 7085.659499] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 7087.407194] Key type lgssc registered [ 7088.573034] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7098.448124] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7098.471798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7099.935636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7104.919475] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7110.127716] LustreError: 214201:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7110.157044] LustreError: 214201:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7115.248736] LustreError: 214202:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7116.773518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7117.342979] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7123.409209] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7132.557728] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7140.455985] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7149.621941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7150.137393] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7157.142179] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7165.726621] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7169.593846] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 7189.987338] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7189.991204] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7190.015930] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7190.033576] Lustre: Skipped 1 previous similar message [ 7195.107962] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7195.124911] Lustre: Skipped 1 previous similar message [ 7195.666348] Lustre: server umount lustre-OST0000 complete [ 7200.225885] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7200.227049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7200.240719] Lustre: Skipped 2 previous similar messages [ 7200.252385] Lustre: Skipped 1 previous similar message [ 7205.344367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7205.362495] Lustre: Skipped 1 previous similar message [ 7206.104432] Lustre: server umount lustre-MDT0000 complete [ 7209.981617] LustreError: 214182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448062 with bad export cookie 3481062259155237487 [ 7210.001858] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7210.374847] Lustre: server umount lustre-MDT0001 complete [ 7226.026692] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 7229.242443] Key type lgssc unregistered [ 7229.524222] LNet: 217914:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7229.540336] LNetError: 217914:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7230.567639] LNet: Removed LNI 192.168.202.116@tcp [ 7231.669089] Key type .llcrypt unregistered [ 7231.671558] Key type ._llcrypt unregistered [ 7250.769735] Key type ._llcrypt registered [ 7250.771874] Key type .llcrypt registered [ 7250.930910] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 7252.013355] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7252.140184] alg: No test for adler32 (adler32-zlib) [ 7253.657731] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 7253.909624] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 7255.543613] Key type lgssc registered [ 7257.090355] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7266.558910] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7266.616128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7268.155299] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7273.305283] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7277.537624] LustreError: 219182:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7277.553256] LustreError: 219182:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7282.666168] LustreError: 219183:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7282.852892] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7283.568467] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7288.134935] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7296.871701] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7304.087473] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7313.814414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7314.601871] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7324.012815] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7333.650241] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7338.099705] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 7348.276442] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7348.334795] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7348.545505] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7355.190874] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7363.944792] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7365.933104] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7377.732213] Lustre: Permanently deactivating lustre-OST0001 [ 7377.747350] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7384.660744] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 7394.184955] Lustre: Permanently reactivating lustre-OST0001 [ 7394.204544] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7394.215839] Lustre: Skipped 2 previous similar messages [ 7401.558371] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7401.578574] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7401.589385] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7401.607320] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7401.683675] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7404.978938] Lustre: server umount lustre-OST0001 complete [ 7407.074540] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7407.074735] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7407.081208] LustreError: Skipped 1 previous similar message [ 7407.082801] LustreError: 221004:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7407.088751] Lustre: Skipped 2 previous similar messages [ 7407.134418] LustreError: 221004:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7410.149233] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7410.163176] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7410.209197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7412.194467] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7412.205251] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7412.226525] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7412.715585] LustreError: 221675:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7412.747951] LustreError: 221675:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7415.315735] Lustre: server umount lustre-OST0000 complete [ 7421.925602] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7421.943592] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7421.960461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7421.963333] Lustre: Skipped 1 previous similar message [ 7425.784118] Lustre: server umount lustre-MDT0000 complete [ 7428.080915] LustreError: 219183:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7429.763344] LustreError: 219162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448282 with bad export cookie 8464077467276646042 [ 7429.764566] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7429.778575] LustreError: 219162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7430.660032] Lustre: server umount lustre-MDT0001 complete [ 7442.982201] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 7446.670228] Key type lgssc unregistered [ 7446.972926] LNet: 224280:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7446.983232] LNetError: 224280:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7447.025176] LNet: Removed LNI 192.168.202.116@tcp [ 7447.958616] Key type .llcrypt unregistered [ 7447.960906] Key type ._llcrypt unregistered [ 7496.761781] Key type ._llcrypt registered [ 7496.763511] Key type .llcrypt registered [ 7496.943422] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 7498.023691] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7498.164441] alg: No test for adler32 (adler32-zlib) [ 7499.291919] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 7499.515701] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 7501.319184] Key type lgssc registered [ 7502.738775] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7511.158727] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7511.172769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7512.519378] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7512.561250] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7512.840152] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7517.403568] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7527.053232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7527.150249] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7527.181487] Lustre: 226487:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7527.488630] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7532.712310] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7541.646550] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7548.130899] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7555.603955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7555.767357] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7555.932881] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7562.883640] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7563.975076] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 7571.914136] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7575.580580] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7581.780034] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7590.445578] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7600.525677] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7616.789712] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7616.806181] Lustre: Skipped 1 previous similar message [ 7634.405378] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7634.409841] Lustre: Skipped 1 previous similar message [ 7672.747201] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7672.753582] Lustre: Skipped 4 previous similar messages [ 7742.196646] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7742.204976] Lustre: Skipped 7 previous similar messages [ 7758.304824] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7758.321535] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7758.340288] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7763.492138] Lustre: server umount lustre-OST0000 complete [ 7768.032873] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7768.044166] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7768.057091] Lustre: Skipped 1 previous similar message [ 7768.061646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7768.068463] Lustre: Skipped 1 previous similar message [ 7769.059041] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7769.077659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7769.082979] Lustre: Skipped 1 previous similar message [ 7773.760950] Lustre: server umount lustre-MDT0000 complete [ 7777.544716] LustreError: 229605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448630 with bad export cookie 16362981104483068696 [ 7777.556951] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7778.046481] Lustre: server umount lustre-MDT0001 complete [ 7793.495392] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 7797.005672] Key type lgssc unregistered [ 7797.468654] LNet: 231273:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7797.477508] LNetError: 231273:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7798.505440] LNet: Removed LNI 192.168.202.116@tcp [ 7800.246684] Key type .llcrypt unregistered [ 7800.248985] Key type ._llcrypt unregistered [ 7828.608894] Key type ._llcrypt registered [ 7828.613688] Key type .llcrypt registered [ 7828.728758] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 7830.187968] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7830.248367] alg: No test for adler32 (adler32-zlib) [ 7831.381502] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 7831.757815] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 7833.511800] Key type lgssc registered [ 7835.056729] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7843.070465] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7843.090159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7844.669831] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7849.367563] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7855.073945] LustreError: 232689:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7855.085532] LustreError: 232689:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7859.307687] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7859.674269] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7864.218367] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7872.067035] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7879.040357] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7886.146684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7886.444046] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7889.525996] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7892.740872] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 7901.021862] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7920.608463] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7920.610540] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7920.625136] LustreError: Skipped 1 previous similar message [ 7920.630282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7920.637546] Lustre: Skipped 1 previous similar message [ 7920.666189] Lustre: Skipped 1 previous similar message [ 7923.747267] Lustre: server umount lustre-OST0000 complete [ 7930.852106] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7930.857357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7930.875930] Lustre: Skipped 1 previous similar message [ 7930.883396] Lustre: Skipped 1 previous similar message [ 7934.420371] Lustre: server umount lustre-MDT0000 complete [ 7935.970667] LustreError: 233417:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7935.980802] LustreError: 233417:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7938.368675] LustreError: 232672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448791 with bad export cookie 11912530475932911181 [ 7938.387523] LustreError: 232672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7938.395451] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7938.969208] Lustre: server umount lustre-MDT0001 complete [ 7953.084416] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 7955.989589] Key type lgssc unregistered [ 7956.280776] LNet: 236384:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7956.289791] LNetError: 236384:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7956.309818] LNet: Removed LNI 192.168.202.116@tcp [ 7957.255954] Key type .llcrypt unregistered [ 7957.261833] Key type ._llcrypt unregistered [ 7980.155452] Key type ._llcrypt registered [ 7980.160287] Key type .llcrypt registered [ 7980.278594] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 7981.855558] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7981.873079] alg: No test for adler32 (adler32-zlib) [ 7983.159216] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 7983.444718] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 7985.095521] Key type lgssc registered [ 7986.916781] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8258.251803] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 8262.331344] Key type lgssc unregistered [ 8262.592568] LNet: 238403:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8262.602238] LNetError: 238403:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8262.617790] LNet: Removed LNI 192.168.202.116@tcp [ 8263.698514] Key type .llcrypt unregistered [ 8263.699963] Key type ._llcrypt unregistered [ 8298.494238] Key type ._llcrypt registered [ 8298.501045] Key type .llcrypt registered [ 8298.649697] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 8316.918500] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 8317.996278] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8318.194067] alg: No test for adler32 (adler32-zlib) [ 8319.302042] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 8319.543689] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 8321.183318] Key type lgssc registered [ 8322.712687] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8356.512681] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 8034 sec ========== 11:26:47 (1788449207) [ 8358.655644] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 11:26:49 (1788449209) === [ 8362.904536] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 11:26:53 (1788449213) === [ 8411.430627] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 8414.930086] Key type lgssc unregistered [ 8415.343812] LNet: 243653:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8415.353465] LNetError: 243653:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8415.365847] LNet: Removed LNI 192.168.202.116@tcp [ 8416.188243] Key type .llcrypt unregistered [ 8416.189956] Key type ._llcrypt unregistered