[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 462977270 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002405] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004020] kvm-guest: setup PV IPIs [ 0.006993] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009009] pid_max: default: 32768 minimum: 301 [ 0.011090] LSM: Security Framework initializing [ 0.012069] Yama: becoming mindful. [ 0.014033] SELinux: Initializing. [ 0.015065] *** VALIDATE selinux *** [ 0.024272] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029401] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.031064] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033061] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.034106] *** VALIDATE tmpfs *** [ 0.036245] *** VALIDATE proc *** [ 0.037224] *** VALIDATE cgroup *** [ 0.038009] *** VALIDATE cgroup2 *** [ 0.040169] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.041163] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.042014] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.043029] Spectre V2 : User space: Vulnerable [ 0.044015] Speculative Store Bypass: Vulnerable [ 0.047385] debug: unmapping init [mem 0xffffffffabc59000-0xffffffffabc60fff] [ 0.050141] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.051736] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.052039] ... version: 2 [ 0.053013] ... bit width: 48 [ 0.054015] ... generic registers: 4 [ 0.055018] ... value mask: 0000ffffffffffff [ 0.056018] ... max period: 00007fffffffffff [ 0.057019] ... fixed-purpose events: 3 [ 0.058014] ... event mask: 000000070000000f [ 0.060260] rcu: Hierarchical SRCU implementation. [ 0.062453] smp: Bringing up secondary CPUs ... [ 0.063557] x86: Booting SMP configuration: [ 0.064024] .... node #0, CPUs: #1 #2 #3 [ 0.068231] smp: Brought up 1 node, 4 CPUs [ 0.070021] smpboot: Max logical packages: 1 [ 0.071014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.262450] node 0 deferred pages initialised in 188ms [ 0.266020] devtmpfs: initialized [ 0.267264] x86/mm: Memory block size: 128MB [ 0.269718] gcov: version magic: 0x41383552 [ 0.271156] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.272077] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.273253] pinctrl core: initialized pinctrl subsystem [ 0.275174] [ 0.276010] ************************************************************* [ 0.277021] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.278018] ** ** [ 0.279020] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.280020] ** ** [ 0.281020] ** This means that this kernel is built to expose internal ** [ 0.282023] ** IOMMU data structures, which may compromise security on ** [ 0.283020] ** your system. ** [ 0.284021] ** ** [ 0.285016] ** If you see this message and you are not debugging the ** [ 0.286078] ** kernel, report this immediately to your vendor! ** [ 0.287019] ** ** [ 0.288019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.289019] ************************************************************* [ 0.290882] NET: Registered protocol family 16 [ 0.291556] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.292174] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.293110] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.294549] cpuidle: using governor menu [ 0.297162] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.300535] PCI: Using configuration type 1 for base access [ 0.302126] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.311148] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.314022] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.319121] cryptd: max_cpu_qlen set to 1000 [ 0.322165] ACPI: Added _OSI(Module Device) [ 0.324017] ACPI: Added _OSI(Processor Device) [ 0.326017] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.328018] ACPI: Added _OSI(Processor Aggregator Device) [ 0.333142] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.341438] ACPI: Interpreter enabled [ 0.342082] ACPI: PM: (supports S0 S3 S4 S5) [ 0.343118] ACPI: Using IOAPIC for interrupt routing [ 0.344144] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.345453] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.354803] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.355064] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.356026] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.357127] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.359531] acpiphp: Slot [2] registered [ 0.360113] acpiphp: Slot [5] registered [ 0.361219] acpiphp: Slot [6] registered [ 0.363159] acpiphp: Slot [7] registered [ 0.365163] acpiphp: Slot [8] registered [ 0.367149] acpiphp: Slot [9] registered [ 0.369173] acpiphp: Slot [10] registered [ 0.370157] acpiphp: Slot [3] registered [ 0.372093] acpiphp: Slot [4] registered [ 0.374225] acpiphp: Slot [11] registered [ 0.376162] acpiphp: Slot [12] registered [ 0.378104] acpiphp: Slot [13] registered [ 0.380120] acpiphp: Slot [14] registered [ 0.381129] acpiphp: Slot [15] registered [ 0.383109] acpiphp: Slot [16] registered [ 0.384116] acpiphp: Slot [17] registered [ 0.386108] acpiphp: Slot [18] registered [ 0.387089] acpiphp: Slot [19] registered [ 0.388082] acpiphp: Slot [20] registered [ 0.389101] acpiphp: Slot [21] registered [ 0.391098] acpiphp: Slot [22] registered [ 0.393161] acpiphp: Slot [23] registered [ 0.394120] acpiphp: Slot [24] registered [ 0.396113] acpiphp: Slot [25] registered [ 0.398169] acpiphp: Slot [26] registered [ 0.399111] acpiphp: Slot [27] registered [ 0.401102] acpiphp: Slot [28] registered [ 0.403182] acpiphp: Slot [29] registered [ 0.405143] acpiphp: Slot [30] registered [ 0.406135] acpiphp: Slot [31] registered [ 0.408098] PCI host bridge to bus 0000:00 [ 0.410022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.412027] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.415025] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.418029] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.421025] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.424029] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.426207] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.430095] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.434376] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.445021] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.449584] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.453027] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.455019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.459019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.462524] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.465867] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.469057] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.472836] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.479018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.492022] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.498019] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.504757] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.512023] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.521021] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.536022] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.548176] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.557018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.566017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.585020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.595255] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.603023] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.611021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.629022] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.639657] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.645014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.651017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.668015] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.677833] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.687022] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.696025] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.715022] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.727998] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.734016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.740017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.764017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.779136] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.781524] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.784527] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.787194] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.793389] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.797125] iommu: Default domain type: Passthrough [ 0.799493] SCSI subsystem initialized [ 0.801135] ACPI: bus type USB registered [ 0.802092] usbcore: registered new interface driver usbfs [ 0.804112] usbcore: registered new interface driver hub [ 0.806085] usbcore: registered new device driver usb [ 0.808156] pps_core: LinuxPPS API ver. 1 registered [ 0.809013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.812063] PTP clock support registered [ 0.814224] EDAC MC: Ver: 3.0.0 [ 0.816172] PCI: Using ACPI for IRQ routing [ 0.819248] NetLabel: Initializing [ 0.821017] NetLabel: domain hash size = 128 [ 0.822020] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.825180] NetLabel: unlabeled traffic allowed by default [ 0.827261] vgaarb: loaded [ 0.830285] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.832018] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.838434] clocksource: Switched to clocksource kvm-clock [ 0.943174] VFS: Disk quotas dquot_6.6.0 [ 0.945074] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.947845] *** VALIDATE ramfs *** [ 0.949611] *** VALIDATE hugetlbfs *** [ 0.952470] pnp: PnP ACPI init [ 0.954728] pnp: PnP ACPI: found 6 devices [ 0.984404] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.988975] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.990859] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.993212] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.995376] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.997504] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.999761] NET: Registered protocol family 2 [ 1.001785] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.007377] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.011403] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.016845] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.020472] TCP: Hash tables configured (established 65536 bind 65536) [ 1.023732] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.027152] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.029995] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.033209] NET: Registered protocol family 1 [ 1.036193] RPC: Registered named UNIX socket transport module. [ 1.037679] RPC: Registered udp transport module. [ 1.038971] RPC: Registered tcp transport module. [ 1.041074] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.043311] NET: Registered protocol family 44 [ 1.044779] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.046750] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.048468] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.051055] PCI: CLS 0 bytes, default 64 [ 1.053312] Unpacking initramfs... [ 2.460355] debug: unmapping init [mem 0xffffa056bcc54000-0xffffa056bffbffff] [ 2.466955] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.469868] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.472834] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.985949] Initialise system trusted keyrings [ 2.989420] Key type blacklist registered [ 2.991494] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.002337] zbud: loaded [ 3.005513] *** VALIDATE nfs *** [ 3.006503] *** VALIDATE nfs4 *** [ 3.008315] pstore: using deflate compression [ 3.012970] Platform Keyring initialized [ 3.154142] NET: Registered protocol family 38 [ 3.156244] Key type asymmetric registered [ 3.157869] Asymmetric key parser 'x509' registered [ 3.160607] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.164430] io scheduler mq-deadline registered [ 3.165877] io scheduler kyber registered [ 3.167960] io scheduler bfq registered [ 3.170246] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.173562] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.176400] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.179451] ACPI: Power Button [PWRF] [ 3.185246] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.192080] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.214538] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.226394] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.250377] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.278088] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.310134] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.317247] Non-volatile memory driver v1.3 [ 3.318868] Linux agpgart interface v0.103 [ 3.354871] virtio_blk virtio1: [vda] 149768 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.359574] vda: detected capacity change from 0 to 76681216 [ 3.375202] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.378733] vdb: detected capacity change from 0 to 1073741824 [ 3.395572] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.399175] vdc: detected capacity change from 0 to 2621440000 [ 3.413331] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.416378] vdd: detected capacity change from 0 to 2621440000 [ 3.432373] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.435692] vde: detected capacity change from 0 to 4294967296 [ 3.452194] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.455406] vdf: detected capacity change from 0 to 4294967296 [ 3.461340] libphy: Fixed MDIO Bus: probed [ 3.471879] usbcore: registered new interface driver usbserial_generic [ 3.474231] usbserial: USB Serial support registered for generic [ 3.476333] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.480590] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.482282] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.484636] mousedev: PS/2 mouse device common for all mice [ 3.488292] rtc_cmos 00:05: RTC can wake from S4 [ 3.491679] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.495903] rtc_cmos 00:05: registered as rtc0 [ 3.497743] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.498974] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.504531] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.505538] intel_pstate: CPU model not supported [ 3.512876] hid: raw HID events driver (C) Jiri Kosina [ 3.515355] usbcore: registered new interface driver usbhid [ 3.517418] usbhid: USB HID core driver [ 3.518976] drop_monitor: Initializing network drop monitor service [ 3.521521] Initializing XFRM netlink socket [ 3.523938] NET: Registered protocol family 10 [ 3.527512] Segment Routing with IPv6 [ 3.528760] NET: Registered protocol family 17 [ 3.530488] mpls_gso: MPLS GSO support [ 3.537785] RAS: Correctable Errors collector initialized. [ 3.539735] AVX version of gcm_enc/dec engaged. [ 3.541966] AES CTR mode by8 optimization enabled [ 3.646932] sched_clock: Marking stable (3646895116, 0)->(4671674137, -1024779021) [ 3.651439] registered taskstats version 1 [ 3.654304] Loading compiled-in X.509 certificates [ 3.656219] zswap: loaded using pool lzo/zbud [ 3.685517] Key type big_key registered [ 3.697561] Key type encrypted registered [ 3.699312] ima: No TPM chip found, activating TPM-bypass! [ 3.701131] ima: Allocated hash algorithm: sha1 [ 3.702466] ima: No architecture policies found [ 3.703812] evm: Initialising EVM extended attributes: [ 3.705083] evm: security.selinux [ 3.706039] evm: security.ima [ 3.707165] evm: security.capability [ 3.708721] evm: HMAC attrs: 0x1 [ 3.710940] rtc_cmos 00:05: setting system clock to 2026-09-05 23:40:50 UTC (1788651650) [ 3.718606] debug: unmapping init [mem 0xffffffffacc03000-0xffffffffacdfffff] [ 3.722030] debug: unmapping init [mem 0xffffffffab982000-0xffffffffabc58fff] [ 3.732151] Write protecting the kernel read-only data: 28672k [ 3.737803] debug: unmapping init [mem 0xffffffffaa003000-0xffffffffaa1fffff] [ 3.740940] debug: unmapping init [mem 0xffffffffaa914000-0xffffffffaa9fffff] [ 3.791629] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.802726] systemd[1]: Detected virtualization kvm. [ 3.805379] systemd[1]: Detected architecture x86-64. [ 3.807629] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.860601] systemd[1]: No hostname configured. [ 3.865368] systemd[1]: Set hostname to . [ 3.869635] random: systemd: uninitialized urandom read (16 bytes read) [ 3.874433] systemd[1]: Initializing machine ID from random generator. [ 4.319331] random: systemd: uninitialized urandom read (16 bytes read) [ 4.332654] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 4.347634] random: systemd: uninitialized urandom read (16 bytes read) [ 4.352785] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 4.360916] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Reached target Initrd Root Device. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Sockets. [ OK ] Reached target Timers. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.105870] device-mapper: uevent: version 1.0.3 [ 6.112277] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System.[ 8.150813] random: fast init done [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 8.329620] virtio_net virtio0 ens2: renamed from eth0 [ 8.475081] scsi host0: ata_piix [ 8.562448] scsi host1: ata_piix [ 8.568259] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 8.576959] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 13.235145] random: crng init done [ 13.239896] random: 7 urandom warning(s) missed due to ratelimiting [ 14.499579] dracut-initqueue[584]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 16.584172] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 19.659257] printk: systemd: 26 output lines suppressed due to ratelimiting [ 20.381868] SELinux: Disabled at runtime. [ 20.536643] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 20.555934] systemd[1]: Detected virtualization kvm. [ 20.559854] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 22.084705] systemd[1]: initrd-switch-root.service: Succeeded. [ 22.095597] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 22.105932] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 22.112919] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 22.116133] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 22.135828] systemd[1]: Starting Journal Service... Starting Journal Service... [ 22.158963] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Process Core Dump Socket. Mounting Kernel Debug File System... [ OK ] Stopped target Switch Root. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Control Socket. [ OK [0[ 22.358854] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS m] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Reached target Paths. [ OK ] Stopped target Initrd File Systems. Starting udev Coldplug all Devices... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Remount Root and Kernel File Systems... Mounting Huge Pages File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting POSIX Message Queue File System... [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 23.520769] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 24.421753] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 24.494489] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 25.193366] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 25.283555] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (7s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit)[ 30.247939] Key type dns_resolver registered [ *** ] A start job is running for Configur…-only root support (8s / no limit)[ 31.078947] NFS: Registering the id_resolver key type [ 31.080916] Key type id_resolver registered [ 31.082935] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started OpenSSH server daemon. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg332-server login: [ 50.789308] hrtimer: interrupt took 19297501 ns [ 88.254257] libcfs: loading out-of-tree module taints kernel. [ 88.331550] Key type ._llcrypt registered [ 88.335255] Key type .llcrypt registered [ 88.443738] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_hostid [ 107.710590] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [ 109.372201] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 109.390522] alg: No test for adler32 (adler32-zlib) [ 110.898876] Lustre: Lustre: Build Version: 2.17.58_2_g557ce73 [ 111.872766] LNet: Added LNI 192.168.203.132@tcp [8/256/0/180] [ 113.615297] Key type lgssc registered [ 115.354662] Lustre: Echo OBD driver; http://www.lustre.org/ [ 134.678625] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 179.098564] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [ 192.469644] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 192.512065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 193.756645] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 193.795586] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 193.921764] Lustre: lustre-MDT0000: new disk, initializing [ 194.012526] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 194.032993] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 198.852161] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 212.576630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 212.740111] Lustre: 6514:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 212.817770] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 212.827863] Lustre: Skipped 1 previous similar message [ 212.919255] Lustre: lustre-MDT0001: new disk, initializing [ 212.975828] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 213.010544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 213.031239] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 217.485896] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 222.385910] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 232.333246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 232.593623] Lustre: lustre-OST0000: new disk, initializing [ 232.596508] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 232.601512] Lustre: 8450:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 232.678143] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 238.128152] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 238.140231] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 238.261798] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 238.736670] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 253.101192] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 253.241865] Lustre: lustre-OST0001: new disk, initializing [ 253.249230] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 253.256902] Lustre: 9522:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 253.339467] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 259.090363] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 259.113234] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 259.216664] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 259.541825] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 271.279812] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 278.606847] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 284.818775] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing check_logdir /tmp/testlogs/ [ 289.120756] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing yml_node [ 293.260312] Lustre: DEBUG MARKER: Client: 2.17.58.2 [ 295.269792] Lustre: DEBUG MARKER: MDS: 2.17.58.2 [ 297.312572] Lustre: DEBUG MARKER: OSS: 2.17.58.2 [ 298.878765] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Sep 5 19:45:43 EDT 2026 [ 315.243197] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 316.752099] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 335.843712] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 335.844748] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 335.856297] Lustre: Skipped 2 previous similar messages [ 335.866729] Lustre: Skipped 3 previous similar messages [ 338.509422] Lustre: server umount lustre-MDT0000 complete [ 342.118270] LustreError: 7460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788651988 with bad export cookie 8961281759484644977 [ 342.119306] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 342.145405] LustreError: 7460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 346.080121] LustreError: 10059:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 346.087393] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 346.117048] LustreError: 10059:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 346.175016] Lustre: Skipped 1 previous similar message [ 346.187659] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 346.200477] Lustre: Skipped 1 previous similar message [ 347.123528] LustreError: 6521:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 347.141223] LustreError: 6521:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 348.403515] Lustre: server umount lustre-MDT0001 complete [ 354.243257] Lustre: server umount lustre-OST0000 complete [ 359.755985] Lustre: server umount lustre-OST0001 complete [ 368.391272] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_hostid [ 378.859177] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [ 421.495757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 421.796314] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 421.833213] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 421.965458] Lustre: lustre-MDT0000: new disk, initializing [ 422.113210] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 422.175340] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 427.041306] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 436.944230] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 437.077441] Lustre: 15075:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 437.171190] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 437.182417] Lustre: Skipped 1 previous similar message [ 437.300303] Lustre: lustre-MDT0001: new disk, initializing [ 437.389207] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 437.429775] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 437.440266] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 442.264684] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 451.606741] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 458.817300] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 467.324720] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 467.653219] Lustre: lustre-OST0000: new disk, initializing [ 467.656831] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 467.665238] Lustre: 17045:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 467.791530] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 469.048902] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 469.069878] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 469.290955] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 476.681548] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 488.318733] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 493.966781] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 494.247301] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 499.099557] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 499.338359] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 505.314348] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 505.335105] Lustre: Skipped 1 previous similar message [ 505.354722] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 505.359932] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 505.364383] Lustre: Skipped 2 previous similar messages [ 507.541167] Lustre: server umount lustre-OST0000 complete [ 514.541225] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 514.556180] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 514.574440] Lustre: Skipped 1 previous similar message [ 514.584753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 514.591972] Lustre: Skipped 1 previous similar message [ 517.676731] Lustre: server umount lustre-MDT0000 complete [ 520.686043] LustreError: 15083:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 520.702667] LustreError: 15083:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 520.990940] LustreError: 15067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788652167 with bad export cookie 8961281759484646517 [ 520.995418] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 521.001953] LustreError: 15067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 521.286192] Lustre: server umount lustre-MDT0001 complete [ 524.587752] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 19:49:29 (1788652169) [ 530.182826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 530.574230] LustreError: 19311:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 530.654700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 534.617531] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 536.045899] LustreError: 19312:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 542.341398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 546.897798] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 553.658762] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 559.459952] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 566.127130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 572.160553] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 578.725773] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 586.213726] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 586.214025] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 586.226706] Lustre: Skipped 1 previous similar message [ 586.228289] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 586.248115] Lustre: Skipped 3 previous similar messages [ 591.337444] LustreError: 21133:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 591.351505] LustreError: 21133:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 591.425743] Lustre: server umount lustre-OST0000 complete [ 596.448912] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 596.455019] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 596.457407] Lustre: Skipped 2 previous similar messages [ 596.467291] Lustre: Skipped 1 previous similar message [ 600.606589] Lustre: server umount lustre-MDT0000 complete [ 604.724094] LustreError: 19292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788652251 with bad export cookie 8961281759484647399 [ 604.744419] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 604.751557] LustreError: 19292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 605.357247] Lustre: server umount lustre-MDT0001 complete [ 616.504285] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing unload_modules_local [ 619.196295] Key type lgssc unregistered [ 619.474888] LNet: 22840:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 619.491091] LNetError: 22840:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 620.521232] LNet: Removed LNI 192.168.203.132@tcp [ 621.462197] Key type .llcrypt unregistered [ 621.464452] Key type ._llcrypt unregistered [ 643.164365] Key type ._llcrypt registered [ 643.166654] Key type .llcrypt registered [ 643.258644] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [ 644.137625] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 644.177544] alg: No test for adler32 (adler32-zlib) [ 645.196945] Lustre: Lustre: Build Version: 2.17.58_2_g557ce73 [ 645.451086] LNet: Added LNI 192.168.203.132@tcp [8/256/0/180] [ 647.152350] Key type lgssc registered [ 648.158882] Lustre: Echo OBD driver; http://www.lustre.org/ [ 655.361355] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 655.392253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 656.873220] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 661.099482] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 667.109100] LustreError: 24241:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 667.143706] LustreError: 24241:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 669.487800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 669.909134] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 675.042607] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 684.356207] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 689.721388] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 695.742246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 696.082816] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 701.132509] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 701.233960] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 707.195560] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 719.842188] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 719.850347] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 719.854783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 719.865684] Lustre: Skipped 1 previous similar message [ 719.876530] Lustre: Skipped 1 previous similar message [ 721.639732] Lustre: server umount lustre-OST0000 complete [ 724.961628] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 724.971419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 724.977647] Lustre: Skipped 1 previous similar message [ 724.990780] Lustre: Skipped 1 previous similar message [ 730.080476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 730.088695] Lustre: Skipped 1 previous similar message [ 735.199936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 735.206210] Lustre: Skipped 1 previous similar message [ 739.295115] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 739.296979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 739.307457] Lustre: Skipped 1 previous similar message [ 739.485302] Lustre: server umount lustre-MDT0000 complete [ 742.866456] LustreError: 24222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788652389 with bad export cookie 11560433873868331801 [ 742.873212] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 742.880715] LustreError: 24222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 743.327610] Lustre: server umount lustre-MDT0001 complete [ 753.069405] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing unload_modules_local [ 755.376442] Key type lgssc unregistered [ 755.634866] LNet: 28218:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 755.645277] LNetError: 28218:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 755.672686] LNet: Removed LNI 192.168.203.132@tcp [ 756.291042] Key type .llcrypt unregistered [ 756.293448] Key type ._llcrypt unregistered [ 774.889195] Key type ._llcrypt registered [ 774.892637] Key type .llcrypt registered [ 774.989288] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [ 775.781824] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 775.858339] alg: No test for adler32 (adler32-zlib) [ 776.897741] Lustre: Lustre: Build Version: 2.17.58_2_g557ce73 [ 777.119428] LNet: Added LNI 192.168.203.132@tcp [8/256/0/180] [ 778.759266] Key type lgssc registered [ 779.773554] Lustre: Echo OBD driver; http://www.lustre.org/ [ 786.666887] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 786.685632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 788.106468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 791.843864] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 797.670001] LustreError: 29608:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 797.698246] LustreError: 29608:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 800.042026] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 800.417423] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 805.058607] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 812.996476] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 820.401066] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 838.002742] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 838.309671] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 846.192446] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 854.306271] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 857.846119] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 866.272230] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 866.288671] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 866.316126] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 868.323302] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 868.331933] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 868.334118] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 871.126362] Lustre: server umount lustre-OST0000 complete [ 877.027171] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 877.034082] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 877.048619] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 877.050689] Lustre: Skipped 1 previous similar message [ 883.688474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 883.701057] Lustre: Skipped 3 previous similar messages [ 888.800760] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 888.826810] Lustre: Skipped 1 previous similar message [ 889.311088] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 889.850835] Lustre: server umount lustre-MDT0000 complete [ 893.923177] LustreError: 29607:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 893.946846] LustreError: 29607:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 894.064839] LustreError: 29587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788652540 with bad export cookie 13553600617978893175 [ 894.072344] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 894.075356] LustreError: 29587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 894.469043] Lustre: server umount lustre-MDT0001 complete [ 906.182785] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing unload_modules_local [ 909.649936] Key type lgssc unregistered [ 910.010386] LNet: 33490:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 910.020654] LNetError: 33490:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 910.033822] LNet: Removed LNI 192.168.203.132@tcp [ 911.062276] Key type .llcrypt unregistered [ 911.067432] Key type ._llcrypt unregistered [ 932.869826] Key type ._llcrypt registered [ 932.872478] Key type .llcrypt registered [ 932.978590] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [ 933.958587] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 933.984988] alg: No test for adler32 (adler32-zlib) [ 935.107668] Lustre: Lustre: Build Version: 2.17.58_2_g557ce73 [ 935.293893] LNet: Added LNI 192.168.203.132@tcp [8/256/0/180] [ 936.935198] Key type lgssc registered [ 938.119080] Lustre: Echo OBD driver; http://www.lustre.org/ [ 945.223947] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 945.251775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 946.776515] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 951.698552] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 956.898591] LustreError: 34898:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 956.916448] LustreError: 34898:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 961.304602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 961.745954] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 966.314576] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 973.989553] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 979.919435] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 986.259740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 986.587116] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 991.756552] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 992.870675] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 999.925118] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1012.192864] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1012.214909] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1012.218413] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1012.253136] Lustre: Skipped 1 previous similar message [ 1014.490037] Lustre: server umount lustre-OST0000 complete [ 1022.435658] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1022.441762] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1022.456613] Lustre: Skipped 1 previous similar message [ 1022.472754] Lustre: Skipped 2 previous similar messages [ 1024.932518] Lustre: server umount lustre-MDT0000 complete [ 1027.553171] LustreError: 34894:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1027.581710] LustreError: 34894:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1028.168934] LustreError: 37683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788652674 with bad export cookie 12402492135121511596 [ 1028.174450] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1028.176100] LustreError: 37683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1028.553570] Lustre: server umount lustre-MDT0001 complete [ 1039.182398] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing unload_modules_local [ 1041.847512] Key type lgssc unregistered [ 1042.180476] LNet: 38433:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1042.199170] LNetError: 38433:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1042.221567] LNet: Removed LNI 192.168.203.132@tcp [ 1043.247086] Key type .llcrypt unregistered [ 1043.253283] Key type ._llcrypt unregistered [ 1066.181805] Key type ._llcrypt registered [ 1066.183941] Key type .llcrypt registered [ 1066.260500] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [ 1067.154072] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1067.427695] alg: No test for adler32 (adler32-zlib) [ 1068.438634] Lustre: Lustre: Build Version: 2.17.58_2_g557ce73 [ 1068.726216] LNet: Added LNI 192.168.203.132@tcp [8/256/0/180] [ 1070.383362] Key type lgssc registered [ 1071.370243] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1080.903465] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1080.938071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1082.485411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1087.328821] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 1093.103279] LustreError: 39844:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1093.119538] LustreError: 39844:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1096.055232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1096.434435] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1101.158956] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 1108.219332] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1113.221125] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1120.354200] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1120.786562] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1126.796735] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 1127.993531] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1133.964099] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1141.820440] Lustre: lustre-OST0000: Not available for connect from 192.168.203.32@tcp (stopping) [ 1142.253957] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1142.263927] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1143.264688] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1143.275349] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1143.277377] Lustre: Skipped 1 previous similar message [ 1146.417661] Lustre: server umount lustre-OST0000 complete [ 1168.357808] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1168.365043] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1168.377598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1168.384682] Lustre: Skipped 1 previous similar message [ 1172.098682] Lustre: server umount lustre-MDT0000 complete [ 1175.011260] LustreError: 39843:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1175.030662] LustreError: 39843:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1175.809528] LustreError: 39825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788652822 with bad export cookie 9568290866846324770 [ 1175.822675] LustreError: 39825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1175.831528] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1190.367196] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1190.734585] Lustre: server umount lustre-MDT0001 complete [ 1200.239524] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing unload_modules_local [ 1202.934022] Key type lgssc unregistered [ 1203.223712] LNet: 43552:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1203.242125] LNetError: 43552:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1203.270428] LNet: Removed LNI 192.168.203.132@tcp [ 1204.255598] Key type .llcrypt unregistered [ 1204.258741] Key type ._llcrypt unregistered [ 1228.434401] Key type ._llcrypt registered [ 1228.436830] Key type .llcrypt registered [ 1228.538605] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [ 1229.657204] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1229.695263] alg: No test for adler32 (adler32-zlib) [ 1230.852981] Lustre: Lustre: Build Version: 2.17.58_2_g557ce73 [ 1231.097637] LNet: Added LNI 192.168.203.132@tcp [8/256/0/180] [ 1232.783154] Key type lgssc registered [ 1233.870982] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1242.909394] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1242.937447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1244.592683] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1249.943904] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 1254.887094] LustreError: 44978:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1254.934398] LustreError: 44978:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1260.010193] LustreError: 44977:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1260.778573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1261.072775] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1266.557134] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 1275.129956] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1280.822903] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1287.976164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1288.357671] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1289.400638] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1295.192682] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 1302.859528] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1315.914074] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.32@tcp (stopping) [ 1316.325649] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1316.345618] Lustre: Skipped 2 previous similar messages [ 1319.595345] Lustre: server umount lustre-MDT0000 complete [ 1320.983486] LustreError: 44974:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1323.112419] LustreError: 46823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788652969 with bad export cookie 18312076463464688617 [ 1323.128457] LustreError: 46823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1323.135846] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1326.103909] LustreError: 44973:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1326.123939] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.32@tcp (stopping) [ 1326.131661] LustreError: 44973:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1326.159828] Lustre: Skipped 3 previous similar messages [ 1326.563424] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1328.621530] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1328.625169] Lustre: Skipped 1 previous similar message [ 1329.577782] Lustre: server umount lustre-MDT0001 complete [ 1345.221233] Lustre: server umount lustre-OST0000 complete [ 1361.343364] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing unload_modules_local [ 1364.090375] Key type lgssc unregistered [ 1364.369643] LNet: 48838:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1364.383233] LNetError: 48838:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1364.405485] LNet: Removed LNI 192.168.203.132@tcp [ 1365.335620] Key type .llcrypt unregistered [ 1365.344349] Key type ._llcrypt unregistered [ 1389.081394] Key type ._llcrypt registered [ 1389.083415] Key type .llcrypt registered [ 1389.186334] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [ 1390.452396] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1390.488480] alg: No test for adler32 (adler32-zlib) [ 1391.682509] Lustre: Lustre: Build Version: 2.17.58_2_g557ce73 [ 1391.945666] LNet: Added LNI 192.168.203.132@tcp [8/256/0/180] [ 1393.639940] Key type lgssc registered [ 1394.643233] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1401.784271] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1401.842403] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1417.568517] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1422.687699] LustreError: 50221:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.132@tcp: failed processing log, type 4: rc = -110 [ 1449.375408] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1455.600807] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 1463.047684] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1631.746489] Lustre: server umount lustre-OST0000 complete [ 1648.951751] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing unload_modules_local [ 1651.852229] Key type lgssc unregistered [ 1652.243828] LNet: 51896:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1652.248449] LNetError: 51896:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1652.276386] LNet: Removed LNI 192.168.203.132@tcp [ 1653.159331] Key type .llcrypt unregistered [ 1653.162839] Key type ._llcrypt unregistered [ 1677.814843] Key type ._llcrypt registered [ 1677.819456] Key type .llcrypt registered [ 1677.948797] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [ 1679.257720] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1679.386319] alg: No test for adler32 (adler32-zlib) [ 1680.515946] Lustre: Lustre: Build Version: 2.17.58_2_g557ce73 [ 1680.820460] LNet: Added LNI 192.168.203.132@tcp [8/256/0/180] [ 1682.497813] Key type lgssc registered [ 1684.173616] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1692.390924] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1692.429250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1694.001938] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1699.042415] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 1703.398223] LustreError: 53306:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1703.422078] LustreError: 53306:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1708.438283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1708.513866] LustreError: 53305:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1708.803892] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1713.380830] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 1721.201750] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1726.960891] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1733.868074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1734.423169] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1738.559337] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1740.590024] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 1748.554311] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1754.086657] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1754.090298] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1754.109390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1755.178995] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1760.227809] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1760.251168] Lustre: Skipped 1 previous similar message [ 1765.343418] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1765.351918] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1765.362733] Lustre: Skipped 1 previous similar message [ 1765.510638] Lustre: server umount lustre-OST0000 complete [ 1770.465090] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1770.470373] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1770.472564] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1770.472573] Lustre: Skipped 1 previous similar message [ 1770.483023] Lustre: Skipped 1 previous similar message [ 1776.303501] Lustre: server umount lustre-MDT0000 complete [ 1780.344184] LustreError: 53286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788653427 with bad export cookie 4565831444745525114 [ 1780.353879] LustreError: 53286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1780.365947] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1780.665489] Lustre: server umount lustre-MDT0001 complete [ 1792.066289] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing unload_modules_local [ 1794.996107] Key type lgssc unregistered [ 1795.337179] LNet: 56835:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1795.351667] LNetError: 56835:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1795.384369] LNet: Removed LNI 192.168.203.132@tcp [ 1796.431463] Key type .llcrypt unregistered [ 1796.437277] Key type ._llcrypt unregistered [ 1820.426847] Key type ._llcrypt registered [ 1820.428991] Key type .llcrypt registered [ 1820.503320] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [ 1821.651776] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1821.671739] alg: No test for adler32 (adler32-zlib) [ 1822.713640] Lustre: Lustre: Build Version: 2.17.58_2_g557ce73 [ 1822.946895] LNet: Added LNI 192.168.203.132@tcp [8/256/0/180] [ 1824.615175] Key type lgssc registered [ 1825.546581] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1833.206314] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1833.219984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1848.735961] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1853.856144] LustreError: 58217:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.132@tcp: failed processing log, type 4: rc = -110 [ 1880.543242] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1887.424882] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 1895.987716] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1904.204187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1904.874795] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1904.960577] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1910.252636] LustreError: 58242:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1910.278863] LustreError: 58242:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1910.843103] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 1915.368699] LustreError: 58243:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1915.397754] LustreError: 58243:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1920.384641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1920.487907] LustreError: 59254:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1920.507137] LustreError: 59254:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1920.687089] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1926.009412] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 1933.576530] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1939.882931] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1946.080348] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1946.090282] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1946.120266] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1946.593503] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1946.619043] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1946.638693] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1951.715526] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1956.320670] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1956.739597] Lustre: server umount lustre-OST0000 complete [ 1981.433439] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1981.441899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1981.459163] Lustre: Skipped 1 previous similar message [ 1981.474682] Lustre: Skipped 2 previous similar messages [ 1986.528064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1986.541212] Lustre: Skipped 1 previous similar message [ 1987.894734] Lustre: server umount lustre-MDT0000 complete [ 1991.653824] LustreError: 59260:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1991.680543] LustreError: 59260:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1992.230596] LustreError: 58224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788653639 with bad export cookie 4132037079890829274 [ 1992.236433] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1992.248384] LustreError: 58224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1998.708366] Lustre: server umount lustre-MDT0001 complete [ 2010.367856] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing unload_modules_local [ 2012.878839] Key type lgssc unregistered [ 2013.251592] LNet: 61967:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2013.262894] LNetError: 61967:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2013.297250] LNet: Removed LNI 192.168.203.132@tcp [ 2014.200818] Key type .llcrypt unregistered [ 2014.210288] Key type ._llcrypt unregistered [ 2037.426094] Key type ._llcrypt registered [ 2037.428620] Key type .llcrypt registered [ 2037.528081] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [ 2038.810630] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2038.895371] alg: No test for adler32 (adler32-zlib) [ 2039.936201] Lustre: Lustre: Build Version: 2.17.58_2_g557ce73 [ 2040.160223] LNet: Added LNI 192.168.203.132@tcp [8/256/0/180] [ 2041.871627] Key type lgssc registered [ 2043.015386] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2051.284565] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2051.298898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2052.806819] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2057.301342] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 2062.309789] LustreError: 63381:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2062.339602] LustreError: 63381:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2065.567369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2065.880451] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2070.719274] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 2078.426155] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2083.968049] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2090.589655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2091.026304] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2096.932923] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 2098.230565] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2104.374558] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2123.745906] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2123.753445] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2123.770981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2125.801388] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2125.801548] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2125.806193] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2128.753458] Lustre: server umount lustre-OST0000 complete [ 2132.456782] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2132.462181] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2132.477501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2132.484417] Lustre: Skipped 1 previous similar message [ 2136.032950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2136.037440] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2138.598583] Lustre: server umount lustre-MDT0000 complete [ 2141.159096] LustreError: 64385:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2141.198533] LustreError: 64385:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2141.943043] LustreError: 63361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788653788 with bad export cookie 10657726911600751448 [ 2141.946751] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2141.954884] LustreError: 63361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2142.217716] Lustre: server umount lustre-MDT0001 complete [ 2154.090872] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing unload_modules_local [ 2157.111221] Key type lgssc unregistered [ 2157.497840] LNet: 66931:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2157.506909] LNetError: 66931:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2157.522837] LNet: Removed LNI 192.168.203.132@tcp [ 2158.462585] Key type .llcrypt unregistered [ 2158.464717] Key type ._llcrypt unregistered [ 2185.081813] Key type ._llcrypt registered [ 2185.086431] Key type .llcrypt registered [ 2185.186394] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [ 2185.961155] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2186.081814] alg: No test for adler32 (adler32-zlib) [ 2187.222859] Lustre: Lustre: Build Version: 2.17.58_2_g557ce73 [ 2187.498284] LNet: Added LNI 192.168.203.132@tcp [8/256/0/180] [ 2189.127293] Key type lgssc registered [ 2190.188474] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2198.892944] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 20:17:23 (1788653843) [ 2205.355455] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2205.381053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2206.772968] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2210.889559] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 2216.931229] LustreError: 68630:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2216.950296] LustreError: 68630:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2219.092724] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2219.588946] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2223.504254] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 2230.571552] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2236.593461] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2243.547575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2243.971800] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2249.546403] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 2252.142328] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2256.788214] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2261.744308] Lustre: Failing over lustre-MDT0000 [ 2262.015168] Lustre: server umount lustre-MDT0000 complete [ 2262.496088] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2262.503146] LustreError: 68626:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2262.524081] Lustre: Skipped 2 previous similar messages [ 2262.534189] LustreError: 68626:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2263.621508] LustreError: 68626:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2267.621778] LustreError: 68630:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2267.641751] LustreError: 68630:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2271.480351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2271.602773] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2271.778028] Lustre: *** cfs_fail_loc=135, val=0*** [ 2271.858114] LustreError: 71716:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2271.861463] LustreError: 71716:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.132@tcp: cfg command failed: rc = -2 [ 2271.867388] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2271.867388] [ 2271.882404] LustreError: MGC192.168.203.132@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2271.897239] LustreError: 71705:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2271.906431] LustreError: 71705:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [ 2271.927360] LustreError: 71705:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 2271.938194] LustreError: 71705:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2271.968563] Lustre: server umount lustre-MDT0000 complete [ 2271.970506] LustreError: 71705:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2272.749081] LustreError: 68627:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2272.774335] LustreError: 68627:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2278.918523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2279.064073] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2279.077489] LustreError: 68622:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2279.385516] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2279.465331] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2284.060263] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2284.539513] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2284.573220] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2284.616693] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2285.131201] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 2294.769873] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2294.783549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2294.791475] Lustre: Skipped 1 previous similar message [ 2294.801458] Lustre: Skipped 1 previous similar message [ 2299.801670] Lustre: server umount lustre-OST0000 complete [ 2305.000856] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2305.002361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2305.010444] Lustre: Skipped 1 previous similar message [ 2305.040274] Lustre: Skipped 1 previous similar message [ 2310.115556] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2310.119873] Lustre: Skipped 1 previous similar message [ 2315.234742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2315.242563] Lustre: Skipped 1 previous similar message [ 2318.303126] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2318.545898] Lustre: server umount lustre-MDT0000 complete [ 2320.358309] LustreError: 69679:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2320.370293] LustreError: 69679:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 2322.290397] LustreError: 70473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788653969 with bad export cookie 6574189113805962257 [ 2322.290980] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2322.310019] LustreError: 70473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2322.813220] Lustre: server umount lustre-MDT0001 complete [ 2336.716503] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing unload_modules_local [ 2339.589848] Key type lgssc unregistered [ 2339.976700] LNet: 73495:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2339.988254] LNetError: 73495:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2340.012406] LNet: Removed LNI 192.168.203.132@tcp [ 2341.076492] Key type .llcrypt unregistered [ 2341.081422] Key type ._llcrypt unregistered [ 2366.447286] Key type ._llcrypt registered [ 2366.449253] Key type .llcrypt registered [ 2366.541568] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [ 2367.887438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2367.950620] alg: No test for adler32 (adler32-zlib) [ 2369.089987] Lustre: Lustre: Build Version: 2.17.58_2_g557ce73 [ 2369.348418] LNet: Added LNI 192.168.203.132@tcp [8/256/0/180] [ 2371.135184] Key type lgssc registered [ 2372.406495] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2379.933102] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2379.956394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2381.499566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2385.911847] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 2392.038726] LustreError: 74905:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2392.049852] LustreError: 74905:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2394.734348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2395.174960] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2399.652693] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 2406.938406] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2413.615579] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2420.545345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2420.964390] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2427.125839] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2427.243119] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 2434.884938] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2444.545813] Lustre: Failing over lustre-MDT0000 [ 2444.771073] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2444.782825] LustreError: 74905:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2444.801187] LustreError: 74905:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2444.821634] Lustre: server umount lustre-MDT0000 complete [ 2447.920938] LustreError: 74901:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2453.010663] LustreError: 77072:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2453.025576] LustreError: 77072:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2454.116799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2454.204570] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2454.335185] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2454.448122] LustreError: 77992:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2454.461108] LustreError: 77992:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.132@tcp: cfg command failed: rc = -9 [ 2454.471970] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2454.471970] [ 2454.486580] LustreError: MGC192.168.203.132@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2454.515066] LustreError: 77980:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2454.524566] LustreError: 77980:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -9 [ 2454.530750] LustreError: 77980:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 2454.537389] LustreError: 77980:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2454.558510] Lustre: server umount lustre-MDT0000 complete [ 2454.561270] LustreError: 77980:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2458.129633] LustreError: 77396:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2458.151481] LustreError: 77396:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2460.842848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2461.066781] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2461.395072] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2461.453896] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2463.272482] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2466.176783] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 2466.804357] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2466.838208] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2466.906132] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2475.497518] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2475.510669] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2475.527836] Lustre: Skipped 2 previous similar messages [ 2475.550856] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2477.036249] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2477.037886] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2477.066208] Lustre: Skipped 1 previous similar message [ 2480.638750] Lustre: server umount lustre-OST0000 complete [ 2487.265887] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2487.267694] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2487.295488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2487.309328] Lustre: Skipped 1 previous similar message [ 2490.713601] Lustre: server umount lustre-MDT0000 complete [ 2492.385990] LustreError: 75633:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2492.411691] LustreError: 75633:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2494.747806] LustreError: 74886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788654141 with bad export cookie 5265783083175196648 [ 2494.762961] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2494.765825] LustreError: 74886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2495.101584] Lustre: server umount lustre-MDT0001 complete [ 2508.571982] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing unload_modules_local [ 2511.817899] Key type lgssc unregistered [ 2512.081833] LNet: 79773:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2512.087124] LNetError: 79773:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2512.104970] LNet: Removed LNI 192.168.203.132@tcp [ 2513.198786] Key type .llcrypt unregistered [ 2513.205222] Key type ._llcrypt unregistered [ 2526.420626] Key type ._llcrypt registered [ 2526.422795] Key type .llcrypt registered [ 2526.518319] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing unload_modules_local [ 2526.824294] Key type .llcrypt unregistered [ 2526.829927] Key type ._llcrypt unregistered [ 2550.871969] Key type ._llcrypt registered [ 2550.876762] Key type .llcrypt registered [ 2550.992709] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [ 2552.299968] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2552.405582] alg: No test for adler32 (adler32-zlib) [ 2553.460494] Lustre: Lustre: Build Version: 2.17.58_2_g557ce73 [ 2553.704889] LNet: Added LNI 192.168.203.132@tcp [8/256/0/180] [ 2555.415160] Key type lgssc registered [ 2556.521891] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2564.177868] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2564.194436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2565.679599] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2570.467449] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 2575.848052] LustreError: 81666:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2575.872162] LustreError: 81666:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2578.766480] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2579.204407] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2583.802439] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 2592.129488] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2599.246128] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2605.781122] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2606.273834] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2610.353634] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2611.539383] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 2618.364143] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2634.213684] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2634.246347] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2637.744986] Lustre: server umount lustre-OST0000 complete [ 2644.450771] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2644.498161] Lustre: Skipped 1 previous similar message [ 2644.516694] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2644.528342] Lustre: Skipped 2 previous similar messages [ 2647.436556] Lustre: server umount lustre-MDT0000 complete [ 2649.572685] LustreError: 84156:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2649.591102] LustreError: 84156:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2650.899497] LustreError: 81648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788654297 with bad export cookie 7060103666846142253 [ 2650.901438] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2650.908827] LustreError: 81648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2651.160613] Lustre: server umount lustre-MDT0001 complete [ 2662.233628] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing unload_modules_local [ 2664.858753] Key type lgssc unregistered [ 2665.138091] LNet: 85219:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2665.155157] LNetError: 85219:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2665.204791] LNet: Removed LNI 192.168.203.132@tcp [ 2666.026899] Key type .llcrypt unregistered [ 2666.030903] Key type ._llcrypt unregistered [ 2688.356210] Key type ._llcrypt registered [ 2688.358720] Key type .llcrypt registered [ 2688.469194] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [ 2689.568249] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2689.669240] alg: No test for adler32 (adler32-zlib) [ 2690.632567] Lustre: Lustre: Build Version: 2.17.58_2_g557ce73 [ 2690.860227] LNet: Added LNI 192.168.203.132@tcp [8/256/0/180] [ 2692.519222] Key type lgssc registered [ 2693.509577] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2700.008454] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2700.030509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2701.563210] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2706.541776] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 2712.034806] LustreError: 86629:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2712.085274] LustreError: 86629:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2716.217325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2716.581346] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2721.715442] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 2728.999587] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2735.343449] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2742.579400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2742.934442] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2747.079884] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2748.938987] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 2756.201060] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2766.825868] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2766.833880] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2766.861641] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2768.695133] Lustre: server umount lustre-OST0000 complete [ 2772.961667] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2772.970478] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2772.989391] Lustre: Skipped 1 previous similar message [ 2772.994880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2772.998445] Lustre: Skipped 1 previous similar message [ 2777.057743] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2777.063848] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2778.963290] Lustre: server umount lustre-MDT0000 complete [ 2782.183610] LustreError: 86625:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2782.207613] LustreError: 86625:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2782.969171] LustreError: 86607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788654429 with bad export cookie 12749692986469067830 [ 2782.977594] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2783.007270] LustreError: 86607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2783.638801] Lustre: server umount lustre-MDT0001 complete [ 2794.228807] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing unload_modules_local [ 2797.066443] Key type lgssc unregistered [ 2797.400990] LNet: 90159:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2797.409139] LNetError: 90159:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2797.446606] LNet: Removed LNI 192.168.203.132@tcp [ 2798.208643] Key type .llcrypt unregistered [ 2798.211871] Key type ._llcrypt unregistered [ 2821.558980] Key type ._llcrypt registered [ 2821.561338] Key type .llcrypt registered [ 2821.647990] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [ 2822.630338] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2822.646057] alg: No test for adler32 (adler32-zlib) [ 2823.688476] Lustre: Lustre: Build Version: 2.17.58_2_g557ce73 [ 2823.941249] LNet: Added LNI 192.168.203.132@tcp [8/256/0/180] [ 2825.623172] Key type lgssc registered [ 2826.478682] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2833.991261] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2834.028577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2835.526426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2839.652253] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 2845.669648] LustreError: 91570:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2845.684495] LustreError: 91570:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2847.349356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2847.609945] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2851.447525] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 2858.039984] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2863.623103] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2870.325223] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2870.728867] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2876.901781] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 2881.025465] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2884.771746] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2896.368081] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2896.373145] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2896.384138] Lustre: Skipped 1 previous similar message [ 2896.390580] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2897.454955] Lustre: server umount lustre-OST0000 complete [ 2901.473508] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2901.481782] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2901.501616] Lustre: Skipped 1 previous similar message [ 2901.507392] Lustre: Skipped 1 previous similar message [ 2906.593826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2906.598759] Lustre: Skipped 2 previous similar messages [ 2911.712674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2911.718039] Lustre: Skipped 1 previous similar message [ 2915.295223] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2915.548145] Lustre: server umount lustre-MDT0000 complete [ 2916.832690] LustreError: 91564:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2916.862719] LustreError: 91564:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2919.084542] LustreError: 91551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788654565 with bad export cookie 3443974964455532744 [ 2919.107083] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2919.407817] Lustre: server umount lustre-MDT0001 complete [ 2930.070452] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing unload_modules_local [ 2933.028812] Key type lgssc unregistered [ 2933.365598] LNet: 95100:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2933.386386] LNetError: 95100:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2933.409976] LNet: Removed LNI 192.168.203.132@tcp [ 2934.377370] Key type .llcrypt unregistered [ 2934.379505] Key type ._llcrypt unregistered [ 2959.013589] Key type ._llcrypt registered [ 2959.020445] Key type .llcrypt registered [ 2959.115811] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [ 2960.437056] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2960.468179] alg: No test for adler32 (adler32-zlib) [ 2961.539891] Lustre: Lustre: Build Version: 2.17.58_2_g557ce73 [ 2961.756975] LNet: Added LNI 192.168.203.132@tcp [8/256/0/180] [ 2963.424735] Key type lgssc registered [ 2964.775932] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2973.130400] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2973.168536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2988.767518] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2993.888095] LustreError: 96483:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.132@tcp: failed processing log, type 4: rc = -110 [ 3020.576059] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3027.603124] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 3035.376426] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3041.063424] Lustre: server umount lustre-OST0000 complete [ 3048.144572] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 20:31:32 (1788654692) [ 3054.216244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3054.747524] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3059.088274] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 3060.200886] LustreError: 98131:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3060.219973] LustreError: 98131:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3064.297616] LustreError: 98132:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3067.244847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3067.621479] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3072.945938] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 3081.039866] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3086.944151] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3094.377154] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3094.898171] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3100.022607] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3101.442388] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 3108.995846] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3125.735192] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3125.747732] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3125.772730] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3127.778080] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3127.786354] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3127.802978] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3130.688958] Lustre: server umount lustre-OST0000 complete [ 3134.433961] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3134.448047] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3134.457612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3134.471688] Lustre: Skipped 1 previous similar message [ 3138.019897] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3138.020336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3138.063504] Lustre: Skipped 1 previous similar message [ 3140.659961] Lustre: server umount lustre-MDT0000 complete [ 3143.138413] LustreError: 98132:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3143.174835] LustreError: 98132:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3144.613709] LustreError: 98113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788654791 with bad export cookie 15010399859944221179 [ 3144.617084] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3144.640698] LustreError: 98113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3145.092659] Lustre: server umount lustre-MDT0001 complete [ 3155.629500] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing unload_modules_local [ 3158.039236] Key type lgssc unregistered [ 3158.355874] LNet: 101824:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3158.373829] LNetError: 101824:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3158.393062] LNet: Removed LNI 192.168.203.132@tcp [ 3159.167137] Key type .llcrypt unregistered [ 3159.170432] Key type ._llcrypt unregistered [ 3202.493972] Key type ._llcrypt registered [ 3202.496082] Key type .llcrypt registered [ 3202.648374] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [ 3204.287101] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3204.322681] alg: No test for adler32 (adler32-zlib) [ 3205.698932] Lustre: Lustre: Build Version: 2.17.58_2_g557ce73 [ 3206.203627] LNet: Added LNI 192.168.203.132@tcp [8/256/0/180] [ 3207.935177] Key type lgssc registered [ 3209.229907] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3216.976362] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3217.010832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3218.498117] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3222.321703] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 3228.643214] LustreError: 104006:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3228.653277] LustreError: 104006:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3229.961162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3230.348575] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3234.765551] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 3242.309825] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3247.273315] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3253.439433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3253.685255] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3259.658822] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 3263.934947] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3266.344670] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3274.211057] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3274.226183] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3274.248247] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3274.255322] Lustre: Skipped 1 previous similar message [ 3279.345370] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3279.352779] Lustre: Skipped 1 previous similar message [ 3280.102599] Lustre: server umount lustre-OST0000 complete [ 3284.449181] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3284.456950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3284.470657] Lustre: Skipped 2 previous similar messages [ 3284.480525] Lustre: Skipped 1 previous similar message [ 3289.571583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3289.577528] Lustre: Skipped 1 previous similar message [ 3289.687469] Lustre: server umount lustre-MDT0000 complete [ 3292.761952] LustreError: 103987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788654939 with bad export cookie 16782489416178921587 [ 3292.768651] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3292.775545] LustreError: 103987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3293.184645] Lustre: server umount lustre-MDT0001 complete [ 3301.314967] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing unload_modules_local [ 3303.407301] Key type lgssc unregistered [ 3303.674973] LNet: 107538:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3303.685726] LNetError: 107538:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3303.708678] LNet: Removed LNI 192.168.203.132@tcp [ 3304.415148] Key type .llcrypt unregistered [ 3304.419126] Key type ._llcrypt unregistered [ 3319.178306] Key type ._llcrypt registered [ 3319.180099] Key type .llcrypt registered [ 3319.310533] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [ 3319.922822] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3320.043410] alg: No test for adler32 (adler32-zlib) [ 3321.088409] Lustre: Lustre: Build Version: 2.17.58_2_g557ce73 [ 3321.324652] LNet: Added LNI 192.168.203.132@tcp [8/256/0/180] [ 3322.999196] Key type lgssc registered [ 3323.905890] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3330.383409] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3330.412839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3345.955850] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3351.071315] LustreError: 108756:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.132@tcp: failed processing log, type 4: rc = -110 [ 3377.759955] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3383.445537] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 3389.957183] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3395.975706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3396.280061] LustreError: MGC192.168.203.132@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3396.300085] LustreError: 109780:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3396.313211] LustreError: 109780:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [ 3396.325199] LustreError: 109780:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 3396.404852] Lustre: server umount lustre-MDT0000 complete [ 3396.408143] LustreError: 109780:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3414.132418] Lustre: server umount lustre-OST0000 complete [ 3422.260411] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_hostid [ 3430.651611] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [ 3462.185717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3462.396088] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3462.415888] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3462.480448] Lustre: lustre-MDT0000: new disk, initializing [ 3462.540728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3462.553762] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3466.662532] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 3476.599216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3476.743118] Lustre: 112570:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3476.800322] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3476.809106] Lustre: Skipped 1 previous similar message [ 3476.932870] Lustre: lustre-MDT0001: new disk, initializing [ 3477.036493] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3477.082052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3477.099387] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3481.071248] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 3489.848842] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3495.183798] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3501.726189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3501.983606] Lustre: lustre-OST0000: new disk, initializing [ 3501.987661] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3501.993433] Lustre: 114536:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3502.078850] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3508.366798] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 3508.791382] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3508.805644] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3508.986456] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3516.893564] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3521.854215] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3522.035394] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3526.654294] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3526.929624] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3529.699247] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3529.710875] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3529.723494] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3534.817758] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3534.825743] Lustre: Skipped 2 previous similar messages [ 3535.052900] Lustre: server umount lustre-OST0000 complete [ 3539.940525] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3539.952796] Lustre: Skipped 1 previous similar message [ 3539.956562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3545.057180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3545.067774] Lustre: Skipped 2 previous similar messages [ 3550.180236] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3550.188902] Lustre: Skipped 1 previous similar message [ 3553.247126] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3553.452593] Lustre: server umount lustre-MDT0000 complete [ 3558.079859] LustreError: 112563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788655204 with bad export cookie 9724850530567896804 [ 3558.095850] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3558.096896] LustreError: 112563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3558.526057] Lustre: server umount lustre-MDT0001 complete [ 3566.132429] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 20:40:10 (1788655210) [ 3569.148784] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3583.451696] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_hostid [ 3590.783106] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [ 3622.957780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3623.314962] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3623.413333] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3623.563187] Lustre: lustre-MDT0000: new disk, initializing [ 3623.729943] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3623.770378] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3629.526407] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 3640.160872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3640.306786] Lustre: 119440:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3640.370843] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3640.375093] Lustre: Skipped 1 previous similar message [ 3640.439147] Lustre: lustre-MDT0001: new disk, initializing [ 3640.522209] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3640.557265] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3640.572797] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3645.510836] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 3655.094276] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3659.738682] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3666.346176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3666.576814] Lustre: lustre-OST0000: new disk, initializing [ 3666.580346] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3666.585915] Lustre: 121406:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3666.664959] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3672.773933] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 3674.699358] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3674.715852] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3674.788633] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3683.979116] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3690.405338] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3690.653170] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3695.620894] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3695.910734] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3700.714604] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3700.718068] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3700.726723] LustreError: Skipped 2 previous similar messages [ 3700.727449] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3700.754941] Lustre: Skipped 2 previous similar messages [ 3700.795190] Lustre: Skipped 1 previous similar message [ 3703.935381] Lustre: server umount lustre-OST0000 complete [ 3710.945138] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3710.963123] Lustre: Skipped 1 previous similar message [ 3713.766422] Lustre: server umount lustre-MDT0000 complete [ 3716.064263] LustreError: 119451:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3716.084015] LustreError: 119451:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3717.218622] LustreError: 119433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788655364 with bad export cookie 9724850530567897686 [ 3717.222760] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3717.228850] LustreError: 119433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3717.614753] Lustre: server umount lustre-MDT0001 complete [ 3725.213528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3725.626829] LustreError: 123621:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3725.741031] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3730.435630] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 3730.923193] LustreError: 123622:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3736.033601] LustreError: 123621:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3739.978676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3744.923599] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 3752.652243] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3757.768332] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3763.377948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3769.078779] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 3775.431663] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3782.079916] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3786.209119] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3786.211100] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3786.217215] LustreError: Skipped 1 previous similar message [ 3786.225123] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3786.225131] Lustre: Skipped 2 previous similar messages [ 3791.698687] Lustre: server umount lustre-OST0000 complete [ 3796.449076] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3796.461859] Lustre: Skipped 1 previous similar message [ 3801.442890] Lustre: server umount lustre-MDT0000 complete [ 3801.576924] LustreError: 123621:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3801.607946] LustreError: 123621:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3805.010335] LustreError: 123601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788655451 with bad export cookie 9724850530567898568 [ 3805.033918] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3805.473832] Lustre: server umount lustre-MDT0001 complete [ 3816.638565] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing unload_modules_local [ 3819.565784] Key type lgssc unregistered [ 3819.861768] LNet: 127248:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3819.874615] LNetError: 127248:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3819.898403] LNet: Removed LNI 192.168.203.132@tcp [ 3820.888122] Key type .llcrypt unregistered [ 3820.890077] Key type ._llcrypt unregistered [ 3839.257228] Key type ._llcrypt registered [ 3839.264251] Key type .llcrypt registered [ 3839.382336] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_hostid [ 3853.974835] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [ 3855.099787] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3855.160270] alg: No test for adler32 (adler32-zlib) [ 3856.220843] Lustre: Lustre: Build Version: 2.17.58_2_g557ce73 [ 3856.512901] LNet: Added LNI 192.168.203.132@tcp [8/256/0/180] [ 3858.208334] Key type lgssc registered [ 3859.167356] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3891.267515] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3891.298683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3892.569360] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3892.595870] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3892.655280] Lustre: lustre-MDT0000: new disk, initializing [ 3892.705633] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3892.719517] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3896.913987] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 3906.443262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3906.601203] Lustre: 130420:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3906.659154] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3906.663163] Lustre: Skipped 1 previous similar message [ 3906.788859] Lustre: lustre-MDT0001: new disk, initializing [ 3906.900438] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3906.952039] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3906.964579] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3911.004978] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 3919.100647] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3924.379921] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3930.595564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3930.812442] Lustre: lustre-OST0000: new disk, initializing [ 3930.817383] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3930.825691] Lustre: 132386:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3930.892013] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3936.461767] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 3938.884232] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3938.899289] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3939.019311] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3945.564101] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3950.283708] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3950.539131] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3954.851304] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3955.054907] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3959.777983] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3959.780679] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3959.811085] Lustre: Skipped 1 previous similar message [ 3959.811426] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3959.836892] Lustre: Skipped 1 previous similar message [ 3963.091120] Lustre: server umount lustre-OST0000 complete [ 3968.480353] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3968.490166] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3968.523337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3970.019434] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3970.020787] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3970.046629] Lustre: Skipped 1 previous similar message [ 3975.137371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3979.235787] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3979.242715] Lustre: Skipped 2 previous similar messages [ 3981.279144] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3981.505097] Lustre: server umount lustre-MDT0000 complete [ 3985.098676] LustreError: 130410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788655631 with bad export cookie 3138382671505637227 [ 3985.104102] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3985.129156] LustreError: 130410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3985.689921] Lustre: server umount lustre-MDT0001 complete [ 3995.708699] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 20:47:20 (1788655640) [ 4001.884307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4002.367529] LustreError: 134800:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4002.386759] LustreError: 134800:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4002.445262] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4007.009899] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 4007.907299] LustreError: 134801:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4013.029637] LustreError: 134800:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4016.683488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4017.211927] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4022.268435] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 4030.282893] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4035.641555] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4037.600488] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4037.604189] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4037.622538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4043.820901] Lustre: server umount lustre-MDT0000 complete [ 4046.821196] LustreError: 135529:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4046.841649] LustreError: 135529:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4046.875310] LustreError: 134780:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788655693 with bad export cookie 3138382671505638109 [ 4046.877802] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4046.885104] LustreError: 134780:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4047.207378] Lustre: server umount lustre-MDT0001 complete [ 4054.683994] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 20:48:19 (1788655699) [ 4061.097260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4076.705352] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4081.823741] LustreError: 137217:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.132@tcp: failed processing log, type 4: rc = -110 [ 4107.551971] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4113.681836] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 4120.146760] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4121.778738] Lustre: server umount lustre-OST0000 complete [ 4128.481917] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 20:49:33 (1788655773) [ 4134.315204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4134.688858] LustreError: 138662:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4134.757290] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4138.373482] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 4145.147674] LustreError: 138663:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4145.184022] LustreError: 138663:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4147.614094] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4153.029796] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 4160.457883] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4165.755489] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4172.354604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4172.790380] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4172.801593] Lustre: Skipped 1 previous similar message [ 4178.489907] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 4185.480309] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4194.273097] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4194.278825] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4194.298338] Lustre: Skipped 1 previous similar message [ 4194.301246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4194.306409] Lustre: Skipped 3 previous similar messages [ 4197.893390] Lustre: server umount lustre-MDT0000 complete [ 4200.421839] LustreError: 138663:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4200.443841] LustreError: 138663:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4201.448479] LustreError: 138643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788655848 with bad export cookie 3138382671505638767 [ 4201.462786] LustreError: 138643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4201.465073] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4201.781502] Lustre: server umount lustre-MDT0001 complete [ 4215.722194] Lustre: server umount lustre-OST0000 complete [ 4229.039962] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 20:51:13 (1788655873) [ 4239.045765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4239.775120] LustreError: 142318:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4239.925342] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4245.976480] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 4256.447516] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4262.959803] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 4272.595906] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4279.841762] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4287.311300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4294.616126] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 4302.873341] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4307.412040] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4310.191952] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4310.788695] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 4315.343451] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4315.565557] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4317.667667] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4317.688501] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4317.708631] Lustre: Skipped 2 previous similar messages [ 4317.715469] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4317.734650] Lustre: Skipped 3 previous similar messages [ 4323.393454] Lustre: server umount lustre-OST0000 complete [ 4328.421831] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4328.430583] LustreError: Skipped 1 previous similar message [ 4333.306870] Lustre: server umount lustre-MDT0000 complete [ 4336.098290] LustreError: 142318:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4336.113889] LustreError: 142318:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4336.891723] LustreError: 142300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788655983 with bad export cookie 3138382671505639845 [ 4336.896212] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4336.908595] LustreError: 142300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4337.231781] Lustre: server umount lustre-MDT0001 complete [ 4345.556835] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 20:53:10 (1788655990) [ 4352.312919] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4367.840358] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4372.959472] LustreError: 146498:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.132@tcp: failed processing log, type 4: rc = -110 [ 4398.687724] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4398.699929] Lustre: Skipped 2 previous similar messages [ 4405.567502] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 4412.867992] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4418.736063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4419.250897] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4423.139696] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 4430.705831] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4435.394173] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 4442.964913] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4448.437628] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4452.793468] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4453.118573] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4458.110587] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4458.432615] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4462.051070] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4462.066585] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4462.090539] Lustre: Skipped 3 previous similar messages [ 4462.098341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4462.105061] Lustre: Skipped 5 previous similar messages [ 4466.900813] Lustre: server umount lustre-MDT0000 complete [ 4470.248107] LustreError: 147534:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4470.288668] LustreError: 147534:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 4470.630388] LustreError: 146504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788656117 with bad export cookie 3138382671505640867 [ 4470.650545] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4470.998463] Lustre: server umount lustre-MDT0001 complete [ 4484.486748] Lustre: server umount lustre-OST0000 complete [ 4492.952185] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 20:55:37 (1788656137) [ 4498.992055] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4514.591641] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4519.711462] LustreError: 150691:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.132@tcp: failed processing log, type 4: rc = -110 [ 4550.910433] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 4557.462979] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4563.873038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4564.470414] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4568.566349] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 4577.272289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4582.510062] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 4590.385780] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4596.820777] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4604.273131] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4604.490862] Lustre: lustre-OST0001: new disk, initializing [ 4604.499272] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4604.510455] Lustre: 153565:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4611.639897] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4611.650184] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4611.784599] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4611.863530] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 4621.361984] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4626.194723] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4626.422237] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4630.681402] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4630.876866] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4632.548606] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4632.574108] Lustre: Skipped 2 previous similar messages [ 4632.586558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4632.593138] Lustre: Skipped 4 previous similar messages [ 4638.439074] Lustre: server umount lustre-OST0000 complete [ 4642.797770] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4657.119238] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4657.368724] Lustre: server umount lustre-OST0001 complete [ 4667.679461] Lustre: server umount lustre-MDT0000 complete [ 4671.676251] LustreError: 150697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788656318 with bad export cookie 3138382671505641819 [ 4671.680401] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4671.686931] LustreError: 150697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 4672.074179] Lustre: server umount lustre-MDT0001 complete [ 4697.823967] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 20:59:02 (1788656342) [ 4699.578874] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4701.481862] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4703.463955] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 20:59:07 (1788656347) [ 4711.095114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4711.445577] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4711.489990] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4711.761397] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4711.775204] Lustre: Skipped 6 previous similar messages [ 4716.495627] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 4725.363372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4725.523304] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4725.593295] Lustre: 157124:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4730.557221] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 4738.786687] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4744.437517] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4753.331451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4753.565833] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4760.519254] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 4761.759423] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4768.480357] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4773.037536] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4773.300425] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4777.772670] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4777.987209] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4782.050682] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4782.055681] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4782.070685] Lustre: Skipped 5 previous similar messages [ 4785.983520] Lustre: server umount lustre-OST0000 complete [ 4808.080035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4816.042338] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 4822.823761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4822.837340] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4822.838466] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4822.861680] Lustre: Skipped 1 previous similar message [ 4824.003074] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4829.979602] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4830.306852] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4835.632815] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4835.906783] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4841.458746] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff98c744fa2000.ost_server_uuid 50 [ 4842.997481] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff98c744fa2000.ost_server_uuid in FULL state after 0 sec [ 4843.181268] LustreError: 157130:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.203.32@tcp) returned error from blocking AST (req@000000002454ae92 x1875541232976000 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 000000002ca2dc36/0x2b8dc6697055dab5 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.32@tcp remote: 0xeb461251dd7b2ff2 expref: 5 pid: 158992 timeout: 4943 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4843.221685] LustreError: lustre-MDT0000: A client on nid 192.168.203.32@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4843.230942] LustreError: 157120:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.203.32@tcp ns: mdt-lustre-MDT0000_UUID lock: 000000002ca2dc36/0x2b8dc6697055dab5 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.32@tcp remote: 0xeb461251dd7b2ff2 expref: 6 pid: 158992 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4853.992537] Lustre: server umount lustre-OST0000 complete [ 4863.947067] Lustre: server umount lustre-MDT0000 complete [ 4863.970498] LustreError: 157134:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4863.988124] LustreError: 157134:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 4867.684105] LustreError: 157115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788656514 with bad export cookie 3138382671505643002 [ 4867.689886] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4867.696277] LustreError: 157115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4874.164274] Lustre: server umount lustre-MDT0001 complete [ 4884.837648] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing unload_modules_local [ 4887.804673] Key type lgssc unregistered [ 4888.098749] LNet: 163249:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4888.109981] LNetError: 163249:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4888.124172] LNet: Removed LNI 192.168.203.132@tcp [ 4888.925925] Key type .llcrypt unregistered [ 4888.929748] Key type ._llcrypt unregistered [ 4911.868274] Key type ._llcrypt registered [ 4911.870431] Key type .llcrypt registered [ 4912.014121] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [ 4913.198109] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4913.274441] alg: No test for adler32 (adler32-zlib) [ 4914.509317] Lustre: Lustre: Build Version: 2.17.58_2_g557ce73 [ 4914.880963] LNet: Added LNI 192.168.203.132@tcp [8/256/0/180] [ 4916.615157] Key type lgssc registered [ 4917.828704] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4925.686422] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4925.704274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4927.172120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4931.568480] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 4936.677385] LustreError: 164657:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4936.701030] LustreError: 164657:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4940.024638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4940.578854] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4945.723457] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 4954.969326] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4961.635738] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4968.873083] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4969.222582] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4971.268970] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 4976.268802] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 4984.459345] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4990.882069] Lustre: Failing over lustre-MDT0000 [ 4990.952386] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4990.953564] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4990.976568] Lustre: Skipped 1 previous similar message [ 4991.185233] Lustre: server umount lustre-MDT0000 complete [ 4992.048822] LustreError: 164654:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4996.067943] LustreError: 164652:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4996.091037] LustreError: 164652:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5001.186902] LustreError: 165647:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5001.207110] LustreError: 165647:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5003.484098] LustreError: 164639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.32@tcp arrived at 1788656650 with bad export cookie 9554516001790754787 [ 5003.499744] LustreError: 164639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5006.304751] LustreError: 164654:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5006.327045] LustreError: 164654:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5010.539359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5010.650768] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5010.869438] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5010.919969] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5015.167791] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 5016.036943] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5016.044738] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5016.959545] Lustre: lustre-MDT0000: Denying connection for new client 3e7c8bb2-bdfe-4b63-85e0-8a518564145b (at 192.168.203.32@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:09 [ 5026.283547] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5026.303298] Lustre: Skipped 1 previous similar message [ 5026.313522] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5026.320537] Lustre: Skipped 3 previous similar messages [ 5031.476870] Lustre: server umount lustre-OST0000 complete [ 5034.895586] LustreError: 168389:0:(ldlm_lib.c:3029:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5034.900184] Lustre: 167738:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5034.908624] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5034.928726] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5034.936979] LustreError: 167738:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000024881789 x1875542342151168/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5034.949696] LustreError: 167738:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5034.969497] LustreError: 167738:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5034.981799] Lustre: lustre-MDT0000: Recovery over after 0:18, of 2 clients 0 recovered and 2 were evicted. [ 5035.342579] Lustre: server umount lustre-MDT0000 complete [ 5036.517831] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5036.547395] Lustre: Skipped 1 previous similar message [ 5036.560292] LustreError: 166598:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5039.386532] LustreError: 164637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788656686 with bad export cookie 9554516001790754885 [ 5039.406915] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5039.419958] LustreError: 163966:0:(import.c:751:ptlrpc_connect_import_locked()) can't connect to a closed import [ 5041.939992] Lustre: server umount lustre-MDT0001 complete [ 5052.774764] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing unload_modules_local [ 5055.513295] Key type lgssc unregistered [ 5055.922213] LNet: 169139:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5055.928754] LNetError: 169139:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5055.975485] LNet: Removed LNI 192.168.203.132@tcp [ 5056.957892] Key type .llcrypt unregistered [ 5056.966101] Key type ._llcrypt unregistered [ 5081.434907] Key type ._llcrypt registered [ 5081.437509] Key type .llcrypt registered [ 5081.582165] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [ 5082.895875] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5083.012768] alg: No test for adler32 (adler32-zlib) [ 5084.190442] Lustre: Lustre: Build Version: 2.17.58_2_g557ce73 [ 5084.520259] LNet: Added LNI 192.168.203.132@tcp [8/256/0/180] [ 5086.271456] Key type lgssc registered [ 5087.822753] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5096.663452] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5096.677532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5098.084902] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5102.882410] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 5108.229949] LustreError: 170546:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5108.262122] LustreError: 170546:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5113.313326] LustreError: 170547:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5115.500590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5116.436784] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5123.164267] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 5131.637786] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5137.307682] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5143.843435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5144.309352] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5150.814780] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 5157.917126] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5167.866266] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5177.312627] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5177.323641] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5177.337990] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5178.350877] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5178.362686] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5178.366714] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5178.388838] Lustre: Skipped 1 previous similar message [ 5180.971554] Lustre: server umount lustre-OST0000 complete [ 5188.065863] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5188.072847] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5188.093734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5193.696888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5193.713411] Lustre: Skipped 2 previous similar messages [ 5198.817064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5198.832289] Lustre: Skipped 1 previous similar message [ 5200.352349] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5202.572987] Lustre: server umount lustre-MDT0000 complete [ 5203.943597] LustreError: 170547:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5203.961185] LustreError: 170547:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5206.646652] LustreError: 170529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788656853 with bad export cookie 701440427210265107 [ 5206.658656] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5206.664931] LustreError: 170529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5207.017332] Lustre: server umount lustre-MDT0001 complete [ 5218.427083] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing unload_modules_local [ 5221.595358] Key type lgssc unregistered [ 5221.984584] LNet: 174100:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5221.990753] LNetError: 174100:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5222.010059] LNet: Removed LNI 192.168.203.132@tcp [ 5222.947394] Key type .llcrypt unregistered [ 5222.952142] Key type ._llcrypt unregistered [ 5257.817575] Key type ._llcrypt registered [ 5257.819763] Key type .llcrypt registered [ 5257.906179] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [ 5258.858994] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5258.976057] alg: No test for adler32 (adler32-zlib) [ 5260.022168] Lustre: Lustre: Build Version: 2.17.58_2_g557ce73 [ 5260.252576] LNet: Added LNI 192.168.203.132@tcp [8/256/0/180] [ 5261.983172] Key type lgssc registered [ 5263.257841] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5270.793879] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5270.822056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5272.300662] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5277.055728] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 5282.787367] LustreError: 175800:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5282.814487] LustreError: 175800:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5285.914970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5286.283374] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5290.616738] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 5297.773770] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5303.526836] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5312.229405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5312.693923] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5317.831650] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5321.288895] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 5330.434439] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5342.690364] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5342.692559] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5342.712294] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5342.723353] Lustre: Skipped 1 previous similar message [ 5342.733251] Lustre: Skipped 1 previous similar message [ 5347.784575] Lustre: server umount lustre-OST0000 complete [ 5352.929216] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5352.930565] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5352.942739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5352.959252] Lustre: Skipped 1 previous similar message [ 5352.978865] Lustre: Skipped 1 previous similar message [ 5358.055089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5358.059891] Lustre: Skipped 1 previous similar message [ 5363.168928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5363.179283] Lustre: Skipped 1 previous similar message [ 5367.263378] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5367.736753] Lustre: server umount lustre-MDT0000 complete [ 5368.292088] LustreError: 175801:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5368.311875] LustreError: 175801:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5371.728857] LustreError: 175780:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788657018 with bad export cookie 14903798908081850971 [ 5371.737384] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5371.740384] LustreError: 175780:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5372.343804] Lustre: server umount lustre-MDT0001 complete [ 5383.980432] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing unload_modules_local [ 5387.110243] Key type lgssc unregistered [ 5387.461057] LNet: 179336:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5387.471622] LNetError: 179336:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5387.491753] LNet: Removed LNI 192.168.203.132@tcp [ 5388.555822] Key type .llcrypt unregistered [ 5388.557523] Key type ._llcrypt unregistered [ 5411.162280] Key type ._llcrypt registered [ 5411.166712] Key type .llcrypt registered [ 5411.261379] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [ 5412.420480] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5412.543661] alg: No test for adler32 (adler32-zlib) [ 5413.673391] Lustre: Lustre: Build Version: 2.17.58_2_g557ce73 [ 5413.953811] LNet: Added LNI 192.168.203.132@tcp [8/256/0/180] [ 5415.615397] Key type lgssc registered [ 5417.138783] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5428.024345] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5428.035165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5429.419720] Lustre: *** cfs_fail_loc=135, val=0*** [ 5429.525748] LustreError: 180858:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5429.534762] LustreError: 180858:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.132@tcp: cfg command failed: rc = -2 [ 5429.548347] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5429.548347] [ 5429.564447] LustreError: MGC192.168.203.132@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5429.589583] LustreError: 180819:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5429.596738] LustreError: 180819:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [ 5429.608551] LustreError: 180819:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 5429.614474] LustreError: 180819:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5429.735578] Lustre: server umount lustre-MDT0000 complete [ 5429.741096] LustreError: 180819:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5436.506046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5437.053068] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5440.948378] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 5446.630959] LustreError: 181229:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5446.669783] LustreError: 181229:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5449.465685] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5449.814258] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5454.787610] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 5463.088191] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5469.347731] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5472.226380] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5472.232132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5472.244496] Lustre: Skipped 1 previous similar message [ 5474.336866] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5474.356103] Lustre: Skipped 2 previous similar messages [ 5479.393944] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5479.406091] Lustre: Skipped 1 previous similar message [ 5484.522754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5484.532981] Lustre: Skipped 1 previous similar message [ 5486.047642] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5486.299929] Lustre: server umount lustre-MDT0000 complete [ 5489.638574] LustreError: 181224:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5489.691963] LustreError: 181224:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5491.031950] LustreError: 181209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788657137 with bad export cookie 11938819638784981666 [ 5491.053015] LustreError: 181209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5491.054951] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5491.795505] Lustre: server umount lustre-MDT0001 complete [ 5506.268235] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing unload_modules_local [ 5509.168951] Key type lgssc unregistered [ 5509.488527] LNet: 183550:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5509.502798] LNetError: 183550:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5509.537577] LNet: Removed LNI 192.168.203.132@tcp [ 5510.474875] Key type .llcrypt unregistered [ 5510.478373] Key type ._llcrypt unregistered [ 5532.241343] Key type ._llcrypt registered [ 5532.245262] Key type .llcrypt registered [ 5532.385576] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing unload_modules_local [ 5532.733611] Key type .llcrypt unregistered [ 5532.735657] Key type ._llcrypt unregistered [ 5551.864127] Key type ._llcrypt registered [ 5551.866116] Key type .llcrypt registered [ 5551.958668] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [ 5552.905059] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5553.126485] alg: No test for adler32 (adler32-zlib) [ 5554.182528] Lustre: Lustre: Build Version: 2.17.58_2_g557ce73 [ 5554.386839] LNet: Added LNI 192.168.203.132@tcp [8/256/0/180] [ 5556.119162] Key type lgssc registered [ 5557.379482] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5565.302857] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5565.350925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5581.023369] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5586.143777] LustreError: 185676:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.132@tcp: failed processing log, type 4: rc = -110 [ 5612.831190] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5618.852924] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 5626.160990] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5632.232902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5632.941836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5632.998681] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5637.631074] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 5638.128281] LustreError: 185701:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5638.158243] LustreError: 185701:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5643.234953] LustreError: 185702:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5645.489747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5645.870954] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5650.090345] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 5657.744617] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5663.409994] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5665.784448] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5676.515169] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5676.532513] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5676.554405] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5678.560369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5678.573544] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5683.687739] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5683.700279] Lustre: Skipped 2 previous similar messages [ 5688.803936] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5688.817915] Lustre: Skipped 1 previous similar message [ 5690.335416] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5690.522629] Lustre: server umount lustre-OST0000 complete [ 5696.993413] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5697.012736] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5697.030650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5699.040967] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5700.580722] Lustre: server umount lustre-MDT0000 complete [ 5704.036472] LustreError: 185682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788657350 with bad export cookie 2027208345009429542 [ 5704.037355] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5704.049421] LustreError: 185682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5704.589675] Lustre: server umount lustre-MDT0001 complete [ 5716.402369] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing unload_modules_local [ 5719.413186] Key type lgssc unregistered [ 5719.723955] LNet: 189643:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5719.734488] LNetError: 189643:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5719.750474] LNet: Removed LNI 192.168.203.132@tcp [ 5720.601849] Key type .llcrypt unregistered [ 5720.604377] Key type ._llcrypt unregistered [ 5744.248495] Key type ._llcrypt registered [ 5744.251025] Key type .llcrypt registered [ 5744.381913] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [ 5745.743756] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5745.789314] alg: No test for adler32 (adler32-zlib) [ 5746.867671] Lustre: Lustre: Build Version: 2.17.58_2_g557ce73 [ 5747.113089] LNet: Added LNI 192.168.203.132@tcp [8/256/0/180] [ 5748.784154] Key type lgssc registered [ 5750.062608] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5758.223898] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5758.247353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5759.720059] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5764.339865] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 5770.221782] LustreError: 191051:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5770.243594] LustreError: 191051:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5773.519224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5773.879944] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5778.681157] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 5787.401052] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5794.895539] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5803.410154] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5803.950328] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5806.034847] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 5810.912381] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 5819.440941] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5825.270756] Lustre: Failing over lustre-MDT0000 [ 5825.854722] Lustre: server umount lustre-MDT0000 complete [ 5827.149123] LustreError: 192446:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5828.579737] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5828.607764] LustreError: 191045:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5828.612609] Lustre: Skipped 1 previous similar message [ 5828.652682] LustreError: 191045:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5832.212353] LustreError: 192446:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5837.332578] LustreError: 192446:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5837.365175] LustreError: 192446:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5844.840238] Lustre: 190363:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788657475/real 1788657475] req@00000000a86855bd x1875543214550528/t0(0) o400->MGC192.168.203.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788657491 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5844.892971] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5847.377579] LustreError: 191045:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5847.429881] LustreError: 191045:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 5847.564303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5855.584903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5855.666676] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5858.846046] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5860.858000] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5860.910400] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 5860.964233] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 5862.030240] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 5866.829990] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 5872.681650] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 5879.778120] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5879.781246] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5879.788101] LustreError: Skipped 1 previous similar message [ 5879.789437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5879.814041] Lustre: Skipped 2 previous similar messages [ 5881.119345] Lustre: server umount lustre-OST0000 complete [ 5886.436197] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5886.437193] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5886.457943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5886.474502] Lustre: Skipped 1 previous similar message [ 5886.497242] Lustre: Skipped 2 previous similar messages [ 5891.558585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5891.568486] Lustre: Skipped 1 previous similar message [ 5892.064321] Lustre: server umount lustre-MDT0000 complete [ 5896.088902] LustreError: 191031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788657542 with bad export cookie 1059818237813100692 [ 5896.097738] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5896.098486] LustreError: 191031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5896.395853] Lustre: server umount lustre-MDT0001 complete [ 5907.418539] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing unload_modules_local [ 5910.411870] Key type lgssc unregistered [ 5910.778737] LNet: 195896:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5910.786254] LNetError: 195896:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5911.846156] LNet: Removed LNI 192.168.203.132@tcp [ 5913.245143] Key type .llcrypt unregistered [ 5913.251728] Key type ._llcrypt unregistered [ 5938.476970] Key type ._llcrypt registered [ 5938.482459] Key type .llcrypt registered [ 5938.603632] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [ 5939.749899] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5939.767071] alg: No test for adler32 (adler32-zlib) [ 5940.965870] Lustre: Lustre: Build Version: 2.17.58_2_g557ce73 [ 5941.447238] LNet: Added LNI 192.168.203.132@tcp [8/256/0/180] [ 5943.199173] Key type lgssc registered [ 5944.599882] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5953.211054] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5953.235916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5955.033558] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5959.529968] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 5964.258605] LustreError: 197309:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5964.280405] LustreError: 197309:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5968.373339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5968.876665] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5973.900362] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 5982.628413] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5989.543775] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5997.170893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5997.638324] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5999.676422] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6005.160390] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 6013.764317] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6018.282322] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6024.180162] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6042.468871] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6054.886848] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6054.893703] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6054.910789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6054.934321] Lustre: Skipped 1 previous similar message [ 6060.002710] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6060.276308] Lustre: server umount lustre-OST0000 complete [ 6066.148172] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6066.162505] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6066.184219] Lustre: Skipped 1 previous similar message [ 6066.191515] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6066.205565] Lustre: Skipped 1 previous similar message [ 6068.194519] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6068.199308] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6068.237401] Lustre: Skipped 1 previous similar message [ 6070.526317] Lustre: server umount lustre-MDT0000 complete [ 6073.313996] LustreError: 198037:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6073.338170] LustreError: 198037:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6075.255635] LustreError: 199845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788657722 with bad export cookie 5241829183991144217 [ 6075.260627] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6075.265373] LustreError: 199845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6075.899976] Lustre: server umount lustre-MDT0001 complete [ 6091.048634] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing unload_modules_local [ 6094.786429] Key type lgssc unregistered [ 6095.194764] LNet: 201170:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6095.211228] LNetError: 201170:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6095.240522] LNet: Removed LNI 192.168.203.132@tcp [ 6096.460912] Key type .llcrypt unregistered [ 6096.463156] Key type ._llcrypt unregistered [ 6118.928747] Key type ._llcrypt registered [ 6118.931962] Key type .llcrypt registered [ 6119.088061] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [ 6120.548693] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6120.815419] alg: No test for adler32 (adler32-zlib) [ 6121.960922] Lustre: Lustre: Build Version: 2.17.58_2_g557ce73 [ 6122.176396] LNet: Added LNI 192.168.203.132@tcp [8/256/0/180] [ 6123.911329] Key type lgssc registered [ 6125.459707] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6136.120077] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6136.145690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6137.716679] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6144.107901] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 6148.071241] LustreError: 202434:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6148.090738] LustreError: 202434:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6153.185680] LustreError: 202433:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6153.711500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6154.151341] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6159.590697] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 6167.395412] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6173.030147] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6180.548698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6181.041281] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6183.101276] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6187.889098] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 6196.248731] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6203.327233] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6212.878702] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6223.922614] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6230.117511] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6244.839495] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6244.846097] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6244.857213] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6244.859797] Lustre: Skipped 1 previous similar message [ 6249.953261] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6249.972424] Lustre: Skipped 2 previous similar messages [ 6255.081757] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6255.088925] Lustre: Skipped 1 previous similar message [ 6256.095188] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6256.294450] Lustre: server umount lustre-OST0000 complete [ 6261.730136] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6261.742261] LustreError: Skipped 1 previous similar message [ 6261.751073] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6261.771941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6265.314782] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6267.370435] Lustre: server umount lustre-MDT0000 complete [ 6270.440862] LustreError: 203394:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6270.463789] LustreError: 203394:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6271.464300] LustreError: 202415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788657918 with bad export cookie 10085269215591055445 [ 6271.472996] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6271.952623] Lustre: server umount lustre-MDT0001 complete [ 6282.579771] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing unload_modules_local [ 6285.721188] Key type lgssc unregistered [ 6285.965612] LNet: 207360:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6285.975295] LNetError: 207360:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6285.997709] LNet: Removed LNI 192.168.203.132@tcp [ 6286.962177] Key type .llcrypt unregistered [ 6286.966628] Key type ._llcrypt unregistered [ 6313.426883] Key type ._llcrypt registered [ 6313.431178] Key type .llcrypt registered [ 6313.566584] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [ 6315.044925] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6315.104408] alg: No test for adler32 (adler32-zlib) [ 6316.390442] Lustre: Lustre: Build Version: 2.17.58_2_g557ce73 [ 6316.624449] LNet: Added LNI 192.168.203.132@tcp [8/256/0/180] [ 6318.295171] Key type lgssc registered [ 6319.649133] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6328.518594] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6328.558451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6330.108658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6335.049897] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 6339.562675] LustreError: 208769:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6339.592960] LustreError: 208769:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6343.597174] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6344.169035] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6350.001911] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 6358.487875] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6365.566365] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6372.989465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6373.549730] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6374.632738] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6381.896852] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 6389.827435] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6404.578662] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6404.592197] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6404.636612] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6409.306866] Lustre: server umount lustre-OST0000 complete [ 6414.820716] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6414.845391] Lustre: Skipped 1 previous similar message [ 6414.852994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6414.859701] Lustre: Skipped 2 previous similar messages [ 6419.541286] Lustre: server umount lustre-MDT0000 complete [ 6419.936405] LustreError: 209500:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6419.947713] LustreError: 209500:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6423.055778] LustreError: 208750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788658069 with bad export cookie 1056964882775387545 [ 6423.069920] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6423.075278] LustreError: 208750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6423.538177] Lustre: server umount lustre-MDT0001 complete [ 6435.355415] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing unload_modules_local [ 6438.445926] Key type lgssc unregistered [ 6438.889627] LNet: 212453:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6438.902639] LNetError: 212453:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6438.923842] LNet: Removed LNI 192.168.203.132@tcp [ 6439.950887] Key type .llcrypt unregistered [ 6439.953402] Key type ._llcrypt unregistered [ 6460.666181] Key type ._llcrypt registered [ 6460.669280] Key type .llcrypt registered [ 6460.772393] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [ 6461.800203] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6461.830620] alg: No test for adler32 (adler32-zlib) [ 6462.870129] Lustre: Lustre: Build Version: 2.17.58_2_g557ce73 [ 6463.254567] LNet: Added LNI 192.168.203.132@tcp [8/256/0/180] [ 6464.967849] Key type lgssc registered [ 6466.598564] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6475.800748] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6475.819504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6477.280363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6481.932621] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 6487.530906] LustreError: 213715:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6487.552142] LustreError: 213715:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6491.212776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6491.707983] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6496.270069] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 6504.639750] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6510.747157] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6518.934797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6519.341926] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6522.424331] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6525.492404] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 6533.677124] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6552.035431] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6552.039273] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6552.041801] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6552.084537] Lustre: Skipped 1 previous similar message [ 6557.172656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6557.194593] Lustre: Skipped 2 previous similar messages [ 6562.282046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6562.292641] Lustre: Skipped 1 previous similar message [ 6564.319408] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6564.665759] Lustre: server umount lustre-OST0000 complete [ 6572.514977] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6572.521860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6572.543608] Lustre: Skipped 1 previous similar message [ 6572.573382] Lustre: Skipped 1 previous similar message [ 6577.111288] Lustre: server umount lustre-MDT0000 complete [ 6577.640077] LustreError: 213715:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6577.660614] LustreError: 213715:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6582.478631] LustreError: 213697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788658229 with bad export cookie 13853854739103413432 [ 6582.497813] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6582.508912] LustreError: 213697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6583.089933] Lustre: server umount lustre-MDT0001 complete [ 6593.665531] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing unload_modules_local [ 6596.284561] Key type lgssc unregistered [ 6596.569886] LNet: 217400:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6596.588349] LNetError: 217400:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6596.613087] LNet: Removed LNI 192.168.203.132@tcp [ 6597.534774] Key type .llcrypt unregistered [ 6597.536644] Key type ._llcrypt unregistered [ 6614.597404] Key type ._llcrypt registered [ 6614.602226] Key type .llcrypt registered [ 6614.710630] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [ 6615.409159] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6615.440053] alg: No test for adler32 (adler32-zlib) [ 6616.560039] Lustre: Lustre: Build Version: 2.17.58_2_g557ce73 [ 6616.829115] LNet: Added LNI 192.168.203.132@tcp [8/256/0/180] [ 6618.495153] Key type lgssc registered [ 6619.761660] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6628.341934] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6628.381910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6629.819879] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6634.246735] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 6640.098517] LustreError: 218661:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6640.117280] LustreError: 218661:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6642.034133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6642.347607] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6646.296898] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 6653.237586] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6659.097585] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6665.550102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6666.009099] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6672.056289] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 6675.263212] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6678.918575] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6686.934590] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6687.000852] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6687.229118] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6692.811561] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 6700.705585] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6702.849563] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6714.379780] Lustre: Permanently deactivating lustre-OST0001 [ 6714.399450] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6721.095503] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6727.298507] Lustre: Permanently reactivating lustre-OST0001 [ 6727.316328] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6727.324138] Lustre: Skipped 2 previous similar messages [ 6730.227813] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6730.236255] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6730.240606] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6730.251559] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6730.287095] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6742.837417] Lustre: server umount lustre-OST0001 complete [ 6743.027438] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6743.030443] LustreError: 220485:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6743.042578] Lustre: Skipped 2 previous similar messages [ 6743.067519] LustreError: 220485:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6747.106655] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6747.111078] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6747.128519] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6748.135146] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6748.138570] LustreError: 221407:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6748.167516] LustreError: 221407:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6753.023227] Lustre: server umount lustre-OST0000 complete [ 6758.368886] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6758.387290] Lustre: Skipped 1 previous similar message [ 6758.390728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6758.394313] Lustre: Skipped 2 previous similar messages [ 6763.490939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6768.610034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6768.631410] Lustre: Skipped 2 previous similar messages [ 6771.167121] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6771.477480] Lustre: server umount lustre-MDT0000 complete [ 6773.729713] LustreError: 219915:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6773.760968] LustreError: 219915:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6775.251851] LustreError: 218640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788658422 with bad export cookie 3225798619819641422 [ 6775.278777] LustreError: 218640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6775.289547] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6775.532857] Lustre: server umount lustre-MDT0001 complete [ 6787.021624] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing unload_modules_local [ 6789.721782] Key type lgssc unregistered [ 6790.059955] LNet: 223726:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6790.075103] LNetError: 223726:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6790.098164] LNet: Removed LNI 192.168.203.132@tcp [ 6790.910890] Key type .llcrypt unregistered [ 6790.913081] Key type ._llcrypt unregistered [ 6830.733614] Key type ._llcrypt registered [ 6830.735752] Key type .llcrypt registered [ 6830.871297] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [ 6832.160977] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6832.229261] alg: No test for adler32 (adler32-zlib) [ 6833.330234] Lustre: Lustre: Build Version: 2.17.58_2_g557ce73 [ 6833.598663] LNet: Added LNI 192.168.203.132@tcp [8/256/0/180] [ 6835.255237] Key type lgssc registered [ 6836.357811] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6844.519810] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6844.560671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6845.931514] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6845.955636] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6846.141045] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6851.267331] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 6860.472830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6860.600440] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6860.623530] Lustre: 225929:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6860.832807] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6865.272866] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 6872.505881] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6877.950423] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6884.598115] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6884.739523] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6884.955817] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6890.691191] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 6891.119349] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 6898.517127] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6901.356483] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6909.263691] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6917.248039] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6921.707897] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6930.327291] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6953.248575] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6953.255454] Lustre: Skipped 2 previous similar messages [ 6994.605711] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6994.613791] Lustre: Skipped 4 previous similar messages [ 7059.101950] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 7059.109207] Lustre: Skipped 8 previous similar messages [ 7065.572070] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7065.584346] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7065.612128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7070.176379] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7070.187108] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7070.211476] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7071.559094] Lustre: server umount lustre-OST0000 complete [ 7075.821050] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7075.825873] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7075.849759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7075.862939] Lustre: Skipped 1 previous similar message [ 7075.870404] Lustre: Skipped 1 previous similar message [ 7080.931039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7080.943642] Lustre: Skipped 2 previous similar messages [ 7081.317412] Lustre: server umount lustre-MDT0000 complete [ 7084.966174] LustreError: 228716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788658731 with bad export cookie 8849039454766726357 [ 7084.972365] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7084.985325] LustreError: 228716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7085.524116] Lustre: server umount lustre-MDT0001 complete [ 7099.373751] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing unload_modules_local [ 7102.950744] Key type lgssc unregistered [ 7103.354922] LNet: 230719:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7103.381216] LNetError: 230719:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7103.414783] LNet: Removed LNI 192.168.203.132@tcp [ 7104.175427] Key type .llcrypt unregistered [ 7104.177506] Key type ._llcrypt unregistered [ 7130.654948] Key type ._llcrypt registered [ 7130.659555] Key type .llcrypt registered [ 7130.765158] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [ 7131.472727] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7131.483960] alg: No test for adler32 (adler32-zlib) [ 7132.512523] Lustre: Lustre: Build Version: 2.17.58_2_g557ce73 [ 7132.782542] LNet: Added LNI 192.168.203.132@tcp [8/256/0/180] [ 7134.455177] Key type lgssc registered [ 7135.659697] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7142.168053] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7142.197690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7143.742348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7147.918075] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 7154.146816] LustreError: 232131:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7154.160713] LustreError: 232131:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7156.017472] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7156.469351] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7161.007543] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 7167.766964] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7172.852331] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7178.949067] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7179.287978] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7185.152090] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_default_debug -1 all [ 7188.555105] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7191.904567] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7206.884317] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7206.891368] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7206.898868] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7207.333133] Lustre: server umount lustre-OST0000 complete [ 7213.029657] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7213.041392] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7213.072433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7214.048068] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7219.171093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7219.180330] Lustre: Skipped 3 previous similar messages [ 7223.269386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7223.278621] Lustre: Skipped 1 previous similar message [ 7226.335705] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7228.386799] LustreError: 234384:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7228.397178] LustreError: 234384:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7228.479451] Lustre: server umount lustre-MDT0000 complete [ 7232.338168] LustreError: 234760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788658879 with bad export cookie 3835080196518963969 [ 7232.344722] LustreError: MGC192.168.203.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7232.354777] LustreError: 234760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7232.826426] Lustre: server umount lustre-MDT0001 complete [ 7244.639837] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing unload_modules_local [ 7247.956759] Key type lgssc unregistered [ 7248.308594] LNet: 235820:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7248.315775] LNetError: 235820:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7248.334621] LNet: Removed LNI 192.168.203.132@tcp [ 7249.349337] Key type .llcrypt unregistered [ 7249.357833] Key type ._llcrypt unregistered [ 7271.585346] Key type ._llcrypt registered [ 7271.590437] Key type .llcrypt registered [ 7271.732581] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [ 7272.657688] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7272.807345] alg: No test for adler32 (adler32-zlib) [ 7273.920859] Lustre: Lustre: Build Version: 2.17.58_2_g557ce73 [ 7274.209304] LNet: Added LNI 192.168.203.132@tcp [8/256/0/180] [ 7275.903870] Key type lgssc registered [ 7277.123973] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7546.682565] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing unload_modules_local [ 7549.672808] Key type lgssc unregistered [ 7549.977371] LNet: 237832:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7549.983654] LNetError: 237832:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7550.013709] LNet: Removed LNI 192.168.203.132@tcp [ 7550.975490] Key type .llcrypt unregistered [ 7550.977338] Key type ._llcrypt unregistered [ 7577.029280] Key type ._llcrypt registered [ 7577.033367] Key type .llcrypt registered [ 7577.150634] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing set_hostid [ 7596.705327] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing load_modules_local [ 7597.883216] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7597.911746] alg: No test for adler32 (adler32-zlib) [ 7599.234594] Lustre: Lustre: Build Version: 2.17.58_2_g557ce73 [ 7599.592856] LNet: Added LNI 192.168.203.132@tcp [8/256/0/180] [ 7601.311262] Key type lgssc registered [ 7602.429809] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7630.537403] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7329 sec ========== 21:47:55 (1788659275) [ 7632.488656] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 21:47:56 (1788659276) === [ 7635.909422] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 21:48:00 (1788659280) === [ 7679.163061] Lustre: DEBUG MARKER: oleg332-server.virtnet: executing unload_modules_local [ 7681.738533] Key type lgssc unregistered [ 7682.037484] LNet: 243065:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7682.045940] LNetError: 243065:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7682.072505] LNet: Removed LNI 192.168.203.132@tcp [ 7683.015416] Key type .llcrypt unregistered [ 7683.021276] Key type ._llcrypt unregistered